Release Notes - Falcon - Version 0.6 - HTML format

Sub-task

  • [FALCON-20] - Remove dependency on custom InMobi DistCp
  • [FALCON-166] - Instance status start and end dates are rigid and inconvenient
  • [FALCON-279] - Add ACL for Cluster Entity
  • [FALCON-325] - Process lineage information for Replication policies
  • [FALCON-462] - Add ACL for process entity
  • [FALCON-463] - Validate Feed ACL only if authorization is enabled
  • [FALCON-464] - Enforce Authorization for REST API
  • [FALCON-466] - REST APIs must add the entity owner as an implicit filter
  • [FALCON-467] - Provide Hooks to enforce Authorization for custom extensions
  • [FALCON-468] - Add User Documentation for this feature
  • [FALCON-470] - Add support for pagination, filter by, etc. to Entity and Instance List API
  • [FALCON-471] - Add support for limits to list, status, and other applicable API
  • [FALCON-472] - Ability to Filter instances list API by a specific status
  • [FALCON-473] - Add ability to filter API by custom fields
  • [FALCON-474] - Add Bulk APIs to drive the dashboard needs
  • [FALCON-484] - Refactor workflow related code into context, listener and Notification
  • [FALCON-485] - Simplify JMS Message Sender/Consumer and use Workflow Context
  • [FALCON-486] - Introduce Workflow Context in Post Processing
  • [FALCON-487] - Introduce Workflow Context in Rerun framework
  • [FALCON-488] - Introduce Workflow Context in Lineage Service
  • [FALCON-492] - Refactor workflow listener implementation init config driven
  • [FALCON-504] - Add ability to filter by tags in REST API
  • [FALCON-512] - API support for Monitoring
  • [FALCON-557] - Add super-user who is authorized for all
  • [FALCON-579] - Lineage breaks if feed.xml doesn't have the date pattern in feed path location
  • [FALCON-594] - Process lineage information for Retention policies
  • [FALCON-614] - Add pipeline element to process entity
  • [FALCON-615] - Add pipleline element to lineage graph
  • [FALCON-635] - Add recipe option in falcon client
  • [FALCON-636] - Add a sample recipe for disaster recovery of hdfs dirs/files
  • [FALCON-640] - Add ability to specify sort order for orderBy param in RestAPI
  • [FALCON-651] - Typos in Falcon Documentation
  • [FALCON-660] - 7 test classes refactored and few of them documented
  • [FALCON-661] - Add list types API
  • [FALCON-662] - Fetch relationships for a given type API
  • [FALCON-674] - General code factored out for ProcessInstance* tests
  • [FALCON-686] - Add audits for read-only API
  • [FALCON-687] - Add hooks for extensions in Audit
  • [FALCON-701] - HadoopUtil and Util classes documented
  • [FALCON-752] - Override the permissions for staging and working dirs with 755
  • [FALCON-753] - Change the ownership for staging dir to user submitting the feed
  • [FALCON-757] - Discontinue support for Hadoop-1.x
  • [FALCON-758] - Discontinue support for Oozie-3.x
  • [FALCON-762] - Support feed listing for file system storage
  • [FALCON-783] - Create branch-0.6 and update version to 0.6-incubating
  • [FALCON-784] - Update release notes in CHANGES.txt in branch and trunk
  • [FALCON-785] - Document upgrade path from 0.5 to 0.6
  • [FALCON-786] - FalconAuditFilter - Arguments in wrong order
  • [FALCON-793] - Rest API for getting the list of entities with pagination and filter
  • [FALCON-795] - Maven Enforcer plugin is throwing error while building Falcon.
  • [FALCON-799] - Falcon Dashboard unusable when server is started with umask 077
  • [FALCON-897] - remove regression code from source release
  • [FALCON-901] - Addon designer should be excluded from 0.6 release
  • [FALCON-903] - Remove SNAPSHOT from version in */pom.xml in branch-0.6

Bug

  • [FALCON-240] - Instance status from CLI on a feed doesn't give the retention details
  • [FALCON-280] - Validate the ACL in Feed entity with the user submitting the entity
  • [FALCON-357] - HCatalog Feed replication: Hive export job fails when table partition contains multiple dated columns
  • [FALCON-430] - Process update with user
  • [FALCON-441] - Lineage capture fails for feeds with multiple instances
  • [FALCON-443] - Process with Hive workflow engine and filesystem input feeds, table output feed fails
  • [FALCON-444] - Logs dir for replication workflow is incorrect and jobs fail with permission issues
  • [FALCON-445] - Propagate hive credentials defined in cluster entity to hive-site.xml
  • [FALCON-446] - Hive Replications fail because of permissions issue
  • [FALCON-456] - Custom DistCp conflict with core DistCp in container classpath fails falcon workflows
  • [FALCON-459] - Lineage resource API fails with NPE for bad query params
  • [FALCON-460] - Concurrent deletion of same entity results in NPE
  • [FALCON-482] - Concurrent requests made using Proxy Oozie client fail
  • [FALCON-483] - Fix the failing test ConfigurationStoreTest.testConcurrentRemoves on jenkins
  • [FALCON-495] - multi source single target feed replication failing in regression
  • [FALCON-496] - Feed Replication with Empty Directories giving error
  • [FALCON-497] - Able to submit feed even though owner of storage specified (location type=data) is different from the ACL owner
  • [FALCON-514] - Falcon CLI giving error when using -file option with -rerun in instance management
  • [FALCON-554] - Extend time functions available in Falcon to support current & last week
  • [FALCON-556] - Feed Replication _SUCCESS is not getting created on target directory.
  • [FALCON-561] - Falcon jenkins failure in EntityManagerJerseyIT
  • [FALCON-566] - update throwing null job id exception
  • [FALCON-577] - Extend time functions added in Falcon 0.6 to support current & last week are giving UnsupportedOperationException
  • [FALCON-581] - merlin: Refactor code for cross product and make it a method
  • [FALCON-582] - Latest changes to LICENSE files results in build failure
  • [FALCON-583] - Post processing is broken in current trunk
  • [FALCON-587] - Don't delete input data in @AfterClass in falcon-regression tests if clean_test_dir=false
  • [FALCON-590] - Update to ACLs added to process is not handled
  • [FALCON-595] - Improvements to DefaultAuthorizationProvider code
  • [FALCON-598] - org.apache.falcon.entity.ProcessHelper throws NullPointerException if the process has no inputs OR no outputs defined
  • [FALCON-610] - Refactoring and documentation updates
  • [FALCON-611] - Post process arg status is in 'FAILED' state always
  • [FALCON-613] - Add first class tag for 'pipelines' for process entity
  • [FALCON-616] - cluster submission should fail when shared libs copy fail
  • [FALCON-619] - ELExp_FutureAndLatestTest stabilization
  • [FALCON-630] - late data rerun for process broken in trunk
  • [FALCON-632] - Refactoring, documentation stuff
  • [FALCON-643] - Tests with zero-output/input scenario amended to match test case
  • [FALCON-644] - Falcon message producer masks errors in Post processing
  • [FALCON-646] - Refactoring, documentation stuff
  • [FALCON-649] - Remove unnecessary validation for Instance start time in FalconCLI
  • [FALCON-650] - Instance list APIs occassionally fail when orderBy set to starttime or endtime
  • [FALCON-652] - EntityUtils tests are failing
  • [FALCON-669] - Missing optional workflow execution listeners configuration results in NPE
  • [FALCON-677] - Feed data and stats path defaults to /tmp/
  • [FALCON-678] - Falcon's default port has changed to 15443
  • [FALCON-693] - Tests with zero inp/outp fixed to use correct workflow, javadocs added
  • [FALCON-694] - Lineage: StringIndexOutOfBoundsException while updating graph DB for replicated instance
  • [FALCON-695] - Lineage: "stored-in" edge is added between feed entity and target cluster
  • [FALCON-702] - JailedFileSystem does not work for relative paths
  • [FALCON-704] - Number of process instances returned in the response are null, when in the request, start and end time specified are the same
  • [FALCON-706] - Parse json with instances to InstancesResult using joda's DateTime
  • [FALCON-707] - ProcessInstanceSuspendTest is failing because input path is not set correctly
  • [FALCON-708] - Falcon assumes a default umask in HDFS for staging dir
  • [FALCON-710] - start/end is optional for lifecycle operations
  • [FALCON-711] - Test retagged
  • [FALCON-712] - Tests fail because oozie process frequency limit have changed
  • [FALCON-713] - UpdateAtSpecificTimeTest#updateTimeInPast_Feed should submitAndSchedule feed rather then submit
  • [FALCON-715] - "end" parameter should be greater than last instance's startTime
  • [FALCON-716] - PrismFeedSuspendTest tests retagged as distributed
  • [FALCON-717] - Shutdown not clean for JMSMessageConsumer
  • [FALCON-718] - For instance/status/process end point checks for start and end time is not working
  • [FALCON-720] - Fix ProcessInstanceStatusTest as per FALCON-166
  • [FALCON-723] - Fix ProcessInstanceResumeTest and ProcessInstanceSuspendTest as per FALCON-166
  • [FALCON-724] - Build fails as Integration test fails
  • [FALCON-726] - Expect less fields in messages in falcon-regression
  • [FALCON-727] - JQuery license is incorrectly mentioned in the header file
  • [FALCON-728] - Unit test failures
  • [FALCON-729] - HCatFeedOperationsTest should not initialize hcat client during var declaration
  • [FALCON-731] - Lineage capture for evicted instance is broken
  • [FALCON-732] - Lineage capture fails for an instance thats not generated by falcon
  • [FALCON-733] - HCatProcess tests fail because _SUCCESS file is non-zero size
  • [FALCON-736] - OneNonCatInputOneHCatOutput test fails because of change in parameter name
  • [FALCON-737] - In NoOutputProcessTest print info before assertion
  • [FALCON-739] - JmsMessageConsumer should start daemon threads
  • [FALCON-740] - Entity kill job calls OozieClient.kill on bundle coord job ids before calling kill on bundle job id
  • [FALCON-746] - Add ACL validation and enforcement tests
  • [FALCON-747] - validateSuccessOnlyStart should use '1' as expected value
  • [FALCON-748] - Falcon throws '413 Full Head' error message when kerberos is enabled with AD
  • [FALCON-750] - Method name fixed
  • [FALCON-759] - Adding cluster entity fails when Yarn TimelineClient is enabled.
  • [FALCON-760] - Messaging is broken for FALCON.ENTITY.TOPIC in case of Eviction
  • [FALCON-764] - Falcon retry of failed process/feed instances broken during restart
  • [FALCON-766] - Falcon workflow rerun by default should rerun only Failed nodes and remove continue option
  • [FALCON-767] - Upgrade Hadoop from 2.4 to 2.5
  • [FALCON-768] - Change dashboard USER_ID to falcon-dashboard
  • [FALCON-773] - Log clean up handlers only work in distributed mode
  • [FALCON-776] - Fixing PrismFeedUpdateTest and FeedInstanceStatus test issues
  • [FALCON-777] - UT intermittent failures in Messaging tests depending on Thread.sleep
  • [FALCON-779] - ProcessInstanceStatusTest#testProcessInstanceStatusReverseDateRange fixed
  • [FALCON-787] - FalconCLI - Submit recipe failed
  • [FALCON-788] - Lineage: Minor bug fixes
  • [FALCON-789] - Post processing is broken for Feeds
  • [FALCON-792] - Integration test EntityManagerPaginationJerseyIT fails occasionally.
  • [FALCON-797] - ProcessInstanceStatusTest#testProcessInstanceStatusEndOutOfRange fixed
  • [FALCON-798] - optionalTest_updateProcessMakeOptionalCompulsory fixed
  • [FALCON-800] - Falcon lineage is not working
  • [FALCON-801] - Increasing time to wait for instance
  • [FALCON-802] - Authorization test disabled
  • [FALCON-803] - Falcon Dashboard : Instances displayed on entity page should be sorted by startTime.
  • [FALCON-811] - falcon superuser should be used instead of admin
  • [FALCON-812] - all the entities must have acl
  • [FALCON-814] - Tests for entities summary API.
  • [FALCON-815] - Rename mapBandwidthKB to mapBandwidth in DistCp
  • [FALCON-819] - Submission of cluster with registry interface fails on secure setup
  • [FALCON-820] - Fix minor nits with HadoopClientFactory
  • [FALCON-824] - Remove usage of Hadoop incompatible API
  • [FALCON-826] - unify staging and working dir requirements
  • [FALCON-827] - Build fails with hadoop-2.4.1
  • [FALCON-829] - Better error message for cluster submission failure
  • [FALCON-831] - Operation on non existing entity throws internal server error
  • [FALCON-832] - more edge labels and node type for lineage
  • [FALCON-833] - FeedReplicationTest.availabilityFlagTest fix
  • [FALCON-835] - Tag authorization tests
  • [FALCON-837] - ListProcessInstancesTest added
  • [FALCON-839] - Authorization succeds with invalid acl owner based on group membership
  • [FALCON-840] - Possible NPE in filteredInstanceSet method of AbstractInstanceManager
  • [FALCON-841] - Test falcon process with different frequencies
  • [FALCON-844] - List instances tests
  • [FALCON-845] - superuser falcon is not able to delete/update entity
  • [FALCON-846] - Cluster submitted by one user when deleted by another leads to internal server error
  • [FALCON-847] - Issue with login from keytab in merlin
  • [FALCON-848] - ListEntitiesTest tagged and checkstyle bugs fixed
  • [FALCON-849] - Metadata Rest API throws "500 : Unauthorized Resource" error
  • [FALCON-850] - Cluster summary UI page results in 400 Bad Request
  • [FALCON-851] - Super user authorization is broken
  • [FALCON-852] - Merlin function testProcessOrderBy() needs to use string ordering instead of enum ordering
  • [FALCON-855] - Fix validation in ProcessInstanceKillsTest#testProcessInstanceKillStartAndEndSame
  • [FALCON-857] - Authorization failure results in internal server error
  • [FALCON-858] - entity/instance Rest API - make keys for filterBy param case insensitive
  • [FALCON-859] - EntitySummary RestAPI : Change pathParam "cluster" to queryParam
  • [FALCON-860] - Add label-types for labels added as part of EntitySummaryTest
  • [FALCON-861] - Add ACL tests for falcon client and ACL update
  • [FALCON-862] - Falcon entity Rest API - filter by tags also returns entities that do not have tags.
  • [FALCON-863] - Fix tests to have start as well as end time for kill, rerun etc requests
  • [FALCON-864] - Falcon superuser is unable to delete scheduled feed
  • [FALCON-865] - Rest end-point entities/summary has changed
  • [FALCON-866] - Lineage endpoints need to be changed for merlin
  • [FALCON-867] - In process ACL tests process validity needs to be in future
  • [FALCON-868] - Rerun command incorrect in falcon CLI documentation
  • [FALCON-874] - Rest API - Instance and Entity orderBy param values should be case insensitive
  • [FALCON-875] - Enitiy Summary endpoint filterBy does not filter entities without pipelines
  • [FALCON-876] - Reducing fail time for HCatProcessTest and PrismFeedReplicationUpdateTest
  • [FALCON-880] - Oozie Java actions for hive tables fail in secure mode
  • [FALCON-882] - falcon prism / server was not having jersey client
  • [FALCON-883] - Fixing testFeedCustomFilter()
  • [FALCON-885] - RequestID is coming as null for all instance API calls
  • [FALCON-886] - Disable tests that are failing
  • [FALCON-888] - Disabling test for special chars in pipeline name and RescheduleProcessInFinalStatesTest
  • [FALCON-889] - Windows azure replication fails with "wasb" as the scheme to an HDFS file system
  • [FALCON-891] - Removing some more invalid acl tests
  • [FALCON-892] - HCatReplication fails in secure setup
  • [FALCON-893] - Add tests for replication to wasb filesystem
  • [FALCON-894] - Cluster submission with hive registry fails in secure setup
  • [FALCON-1026] - Falcon UI to participate in SSO provided by Knox
  • [FALCON-1028] - Falcon Data Replication and Mirroring UI
  • [FALCON-1029] - Falcon HA

New Feature

  • [FALCON-299] - Falcon to return job tracker URLs for the instances of the process specified
  • [FALCON-400] - Add Authorization for Entities
  • [FALCON-722] - Add SLA for processes

Improvement

  • [FALCON-133] - Upgrade to slf4j 1.7.5 and use SLF4J logger
  • [FALCON-145] - Feed eviction be implemented in appropriate Storage implementation
  • [FALCON-327] - Simplify message passing framework
  • [FALCON-353] - enable dry run feature of oozie for schedule and update
  • [FALCON-469] - Enhance REST API to drive interactive user dashboard
  • [FALCON-481] - Simplify process parent workflow
  • [FALCON-576] - Update Installation notes for Mac OS
  • [FALCON-588] - Baselining designer code
  • [FALCON-612] - Create CHANGES.txt for falcon-regression
  • [FALCON-654] - Exclude junit dependency in pom
  • [FALCON-655] - Skip workflow upload if process won't be submitted
  • [FALCON-657] - String datePattern moved to BaseTestClass in falcon-regression
  • [FALCON-665] - Handle message consumption failures in JMSMessageConsumer
  • [FALCON-666] - Add Alerts for unrecoverable failures
  • [FALCON-675] - Request URLS moved from parameters into methods in falcon-regression
  • [FALCON-685] - All actions by user must be audited
  • [FALCON-744] - Document existing falcon data archival to cloud store
  • [FALCON-745] - LogUtil requires fine control over what to dump
  • [FALCON-756] - Discontinue support for EOL components
  • [FALCON-781] - Release falcon 0.6 umbrella
  • [FALCON-809] - Add message from response if assertSucceeded fails
  • [FALCON-813] - Expose job id for running jobs in Falcon
  • [FALCON-828] - Add hadoop version info as part of falcon admin version API
  • [FALCON-834] - Propagate request id in the response to help trace and debug failures in merlin
  • [FALCON-838] - Add support for https in merlin
  • [FALCON-856] - Handle JsonSyntaxException in hitUrl
  • [FALCON-878] - Move falcon html files to HTML5

Test

Task

  • [FALCON-303] - Ability to limit the result set on the "instances status" call
  • [FALCON-804] - Remove Oozie 3.* patch files from Falcon

Edit/Copy Release Notes

The text area below allows the project release notes to be edited and copied to another document.