When creating a new Task item, whether from an existing object's context or from scratch, once a context is applied and removed, the context is still attached to the Task item as can be seen in the Task list view with the Context tab active. In the details view of the Task object the original context applied with NOT be visible, but the Task object will still appear in the Task list view with that original context.
This does not seem to be the cases for subsequent contexts applied after the first has been added/removed. Subsequent contexts can be removed and they do not show in the Task object's properties, nor are they listed in the removed context's grouping in the Task list view.