Home → Support →

support for resque jobs

karl.baum

01 Mar, 2010 08:52 PM via web

Admin Functions

• Edit
• Remove

Any way to get new relic to monitor my resque jobs?

thx.

-karl

1. Support Staff 2 Posted by Ralph Bodenner on 01 Mar, 2010 09:20 PM

   

   Hi Karl,

   We'll work on an official fix, but for now, here's how one user has gotten Resque working:

   http://ononoma.wordpress.com/2010/02/28/newrelic-rpm-with-resque/

   Ralph

   Admin Functions

   • Edit
   • Remove

2. 3 Posted by karl.baum on 01 Mar, 2010 09:41 PM

   

   I saw that today. Any idea on when rpm/resque compatibility will be
   ready?

   Admin Functions

   • Edit
   • Split
   • Remove

3. Support Staff 4 Posted by Bill Kayser on 02 Mar, 2010 02:02 AM

   

   I can incorporate the workaround into the agent in a release later this week, but more general (less hacky) support is not likely to be available until the 2.11 release.
   

   Admin Functions

   • Edit
   • Remove

4. Bill Kayser resolved this discussion on 02 Mar, 2010 02:02 AM.

5. karl.baum re-opened this discussion on 02 Mar, 2010 11:22 AM

6. 5 Posted by karl.baum on 02 Mar, 2010 11:22 AM

   

   Sounds great. Thx!

   Admin Functions

   • Edit
   • Split
   • Remove

7. 6 Posted by karl.baum on 02 Mar, 2010 11:35 AM

   

   I've tried to incorporate the changes from the blog you mentioned, but nothing is appearing in newrelic.

   class DataExtractionJob
   
   
     @queue = :dataextraction
   
     BATCH_SIZE=1000
   
     def self.perform
       Rails.logger.info("DataExtractionJob.perform")
       NewRelic::Control.instance['resque'] = true
         RawMessage.process_all(RawMessage.unprocessed(:limit=>BATCH_SIZE).supported_host.has_body)
     ensure
       Resque.enqueue(DataExtractionJob)
       NewRelic::Agent.shutdown
     end
   
     class << self
       include NewRelic::Agent::Instrumentation::ControllerInstrumentation
       add_transaction_tracer :perform, :name => 'DataExtractionJob/perform'
     end
   end
   

   And these changes in rpm:

     def ensure_worker_thread_started
         return unless control.agent_enabled? && control.monitor_mode? && !@invalid_license
         if !running? && !control['resque']
           launch_worker_thread
           @stats_engine.spawn_sampler_thread
         end
     end
   

   And my method has run 8 times, but I see nothing in newrelic:

   https://rpm.newrelic.com/v2/accounts/6119/applications/86848

   Any ideas?

   thx.
   

   Admin Functions

   • Edit
   • Split
   • Remove

8. Support Staff 7 Posted by Bill Kayser on 02 Mar, 2010 06:10 PM

   

   Try setting agent_enabled: true in the newrelic.yml and let me know what happens.

   Bill

   Admin Functions

   • Edit
   • Remove

9. 8 Posted by karl.baum on 03 Mar, 2010 02:37 AM

   

   Still don't see any stats in new relic.

   thx.
   

   Admin Functions

   • Edit
   • Split
   • Remove

10. Support Staff 9 Posted by Bill Kayser on 03 Mar, 2010 05:22 PM

    

    I'm looking at the application "Agent Configuration for WeShopNetworkDataExtraction # When 'enabled' is turned on, the agent collects performance data"

    Couple of things:

    • Can you upgrade the agent? You are on an older alpha of the 2.10 release.

    • Can you fix the application name? Looks like there's a format problem in the yaml file.

    • It looks like agent_enabled is not set to true in your newrelic.yml. Is that right? If not, how are you setting that value?

    Admin Functions

    • Edit
    • Remove

11. Support Staff 10 Posted by Devorah on 15 Mar, 2010 09:38 PM

    

    Karl,
    Just wanted to check in and see if you're getting the stats you need now. If not, please try Bill's most recent suggestions and let us know whether or not that helps.

    Devorah

    Admin Functions

    • Edit
    • Remove

12. 11 Posted by mick on 18 Mar, 2010 11:33 PM

    

    Any news on when this will be officially supported?

    Admin Functions

    • Edit
    • Split
    • Remove

13. Support Staff 12 Posted by Bill Kayser on 18 Mar, 2010 11:40 PM

    

    I'm working on it. There are a number of issues we had to address, not just with resque but with delayed job and rake task monitoring. I'm hoping to release a beta early next week. Sorry for the delay.

    Admin Functions

    • Edit
    • Remove

14. 13 Posted by mick on 18 Mar, 2010 11:51 PM

    

    Sending the data out to RPM after each job sounds like a bad idea. Resque was built for supporting very long queues of small fast jobs. This would slow down these small jobs quite a bit, if they had to wait for the RPM agent to submit the data. It would probably also put a high load on the RPM servers.

    Admin Functions

    • Edit
    • Split
    • Remove

15. Support Staff 14 Posted by Bill Kayser on 19 Mar, 2010 01:01 AM

    

    That's all true, but each job is forked right? We don't really have much choice. If we don't send it right after the job finishes then the process exits and we lose any data we've collected.

    I guess I'd have to agree with you. If you're using resque with small jobs, then using RPM to monitor it is probably not a good idea--for you or us. I guess I envisioned resque would have been used for longer running jobs, that being a motivation to run the job asynchronously to begin with. But I can see where that may not always be the case.

    I welcome further thoughts.

    Admin Functions

    • Edit
    • Remove

16. 15 Posted by mick on 19 Mar, 2010 03:01 PM

    

    Some of our jobs run for half a second or even as long a a second. Is that a long running job? I don't know, but it is absolutely a long time executing this stuff in the rails request, so there is plenty of motivation. We are soon going to execute hundreds of these jobs every minute, which resque is really good for.

    Here is an idea:

    We know that we have an instance of redis at hand so we use that as a communications layer between the master/parent resque process and the forked/child processes. Resque has an after_fork hook, so where we patch the forked instance of the RPM agent to store the collected RPM data in redis. We patch the RPM agent running in the mater/parent resque process to pick up tha RPM data from resque, and send it to RPM every minute. This could maybe be done in the before_first_fork hook.

    Admin Functions

    • Edit
    • Split
    • Remove

17. Support Staff 16 Posted by Bill Kayser on 21 Mar, 2010 12:20 AM

    

    Mick if I understand your suggestion, you are proposing that when the forked job is done performing, it flushes the metric data, transaction traces and errors into redis. Then, once a minute the master process could pull the data out of redis and send it to RPM.

    I think this is feasible but it sounds pretty ambitious. You'd have to harvest the data like normal, but instead of sending it to RPM you would serialize it and store it in Redis. In the master process you'd have to add the ability to pull the metric data out of the external source (redis) and combine it with what's in memory. In the end what you'd save is the difference between a post to a remote server and a 'post' of data to redis. I'm afraid it's not really a configuration we could support reliably even if we wanted to invest the time into that kind of enhancement.

    If you did want to take a crack at forking the agent and rpm_contrib gems and working on this, I'd be happy to advise you.
    

    Wish I had better advice for you.
    

    Admin Functions

    • Edit
    • Remove

18. 17 Posted by mick on 21 Mar, 2010 12:35 AM

    

    I'm absolutely going to try implementing my solution, so a few pointers would be much appreciated.

    Thanks

    Admin Functions

    • Edit
    • Split
    • Remove

19. Support Staff 18 Posted by Bill Kayser on 21 Mar, 2010 03:26 PM

    

    Sure, I'll send you some pointers.

    Admin Functions

    • Edit
    • Remove

20. Bill Kayser resolved this discussion on 03 Apr, 2010 01:53 AM.

Comments are currently closed for this discussion. You can start a new one.