Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-1316

JobTracker holds stale references to retired jobs via unreported tasks

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Blocker
    • Resolution: Fixed
    • None
    • 0.21.0
    • jobtracker
    • None
    • Reviewed
    • Hide
      JobTracker holds stale references to TaskInProgress objects and hence indirectly holds reference to retired jobs resulting into memory leak. Only task-attempts which are yet to report their status are left behind in the memory. All the task-attempts are now removed from the JobTracker by iterating over the scheduled task-attempt ids instead of iterating over available task statuses.
      Show
      JobTracker holds stale references to TaskInProgress objects and hence indirectly holds reference to retired jobs resulting into memory leak. Only task-attempts which are yet to report their status are left behind in the memory. All the task-attempts are now removed from the JobTracker by iterating over the scheduled task-attempt ids instead of iterating over available task statuses.

    Description

      JobTracker fails to remove unreported tasks' mapping from taskToTIPMap if the job finishes and retires. Unreported tasks refers to tasks that were scheduled but the tasktracker did not report back with the task status. In such cases a stale reference is held to TaskInProgress (and thus JobInProgress) long after the job is gone leading to memory leak.

      Attachments

        1. mapreduce-1316-v1.11.patch
          14 kB
          Amar Kamat
        2. mapreduce-1316-v1.13-branch20-yahoo.patch
          21 kB
          Amar Kamat
        3. mapreduce-1316-v1.14.1-branch20-yahoo.patch
          38 kB
          Amar Kamat
        4. mapreduce-1316-v1.14-branch20-yahoo.patch
          37 kB
          Amar Kamat
        5. mapreduce-1316-v1.15.patch
          32 kB
          Amar Kamat
        6. mapreduce-1316-v1.15-branch-0.21.patch
          32 kB
          Amar Kamat
        7. mapreduce-1316-v1.15-branch20-yahoo.patch
          38 kB
          Arun Murthy
        8. mapreduce-1316-v1.7.patch
          15 kB
          Amar Kamat
        9. mapreduce-1316-y20s.patch
          38 kB
          Hemanth Yamijala

        Issue Links

          Activity

            People

              amar_kamat Amar Kamat
              amar_kamat Amar Kamat
              Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: