I'd like to propose changing this title to "Add Delete Capability to Collection's Objects". I'm curious to see what discussion uncovers about this.

My personal thought is that this is a necessary-for-completeness and simple-to-create capability for Collections. My feeling is that the expediency of this should be determined by the willingness of sponsorship. If people want to sponsor it, let's move forward. If not, let's wait until enough orgs want to sponsor it.

I see this as a necessary addition.

This is needed to support removed data that was accidentally added, because you don't have permission to (TLP or copyright).

Per Terry's question on the call:
Permissions on who is allowed to remove content should be handled outside the the spec and be an implementation detail. We don't control how people have access to what resources via authentication, this is similar. But there should not be a mandated restriction that only the SDO's author is allowed to remove an object.

We discussed this on the working call on 2018-02-20 with the following 15 TC members (Bret Jordan, Trey Darley, Chris Ricard, Christian Hunt, Dan Dye, Dave Lemire, Drew Varner, Efrain Ortiz, Gary Katz, Jane Ginn, John-Mark Gurney, John Wunder, Nicholas Hayden, Paul Patrick, Rich Piazza). There were several comments made by Terry and JMG, but all in all, it seems like everyone is generally in agreement with doing this. However, we decided that we would bring it up again for further discussion.

I am just going to point out that if we add this, just because you issue a DELETE for a specific collection, that doesn't mean or imply it's actually been deleted from the server. A STIX object could be in many collections at the same time - or, could be known to the server, but not present in any collections at all.

We discussed this on the working call on 2018-05-22 and there was broad consensus to add this to the specification. Several concerns were brought up on the call that will need to be addressed in the specification in the form of implementation guidance.

From Jeff Mates to Everyone: (01:29 PM)
in any collection or a specific one?
From Allan Thomson to Everyone: (01:30 PM)
all operations are done in the context of a collection(s). delete operation should be consistent
From Jeff Mates to Everyone: (01:30 PM)
also how does it verify permissions on objects?
From Allan Thomson to Everyone: (01:32 PM)
yes
a non-publisher can’t delete someone else’s content
From Allan Thomson to Everyone: (01:33 PM)
an admin role may be implemented and is a product choice where that admin can do whatever they want. i dont believe that is necessary to add to the taxii server spec
From jmg to Everyone: (01:33 PM)
IMO, who posted/published to the collection doesn't matter.. if the TAXII server wants to allow any random person to delete (depending upon permissions it decides), it should allow it.

This is indeed sticky.

If you delete an object that's been revoked, it can be re-written without revocation as part of a race condition, assuming the Object delete would delete all versions, including revocation. It may be that we want the operation to delete all versions, but leave a simple revocation marker (tombstone) in the database to indicate that the object existed.

However, maybe the delete call is trying to "undo" the revocation by specifically deleting the version of an object that sets revocation?

I get the need to prevent spillage of sensitive information. It's just a pain to implement. This is partially caused by the disconnect between globally scoped STIX objects and the way the endpoint makes it appear that you are posting the STIX objects to a collection when you're really publishing the STIX object to the server that allows Collections to read the objects from the global store.

I added the basic parts of this endpoint to section 5.7.

• "a non-publisher can’t delete someone else’s content". We don't track owner of a collection though in it's metadata (even though any product is likely doing that). So who has the right to remove something from a collection? The creator of the object itself?

• What does this action actually do? Does it just remove the ability to retrieve that object from the collection? What about relationships inside the collection that point at it, do those stay, or become orphaned?

• Many products won't be able to support this feature (all those with READ-only kinds of collections). The only product category likely to support it is those with a repository. Another question I have is if a product that has a WRITE-only collection needs to support this feature. It is not clear to me if it would.

This feature makes me queasy. It falls into the same bucket as the request to put BLOBs into TAXII. Right now I am reserving judgement until more of the proposal is fleshed out.

We need to be very careful, our mandate is to facilitate the sharing of intelligence between any two cybersecurity products, not try to make a end-to-end implementation to build an entire TIP around.

I support the concept of a DELETE operation, but I agree w/ @JasonKeirstead that it's very tricky. I at first was thinking of leaving it up to implementation defined behavior, but at a minimum just remove it from the local collection, but in reality, it's much more complicated.

It sounds like there are multiple use cases, and they should NOT be put/solved by a single implementation.

1. I want to remove it from this, and only this collection.

2. I published sensitive data, I want it removed, and do not want it ever to be visible again. This is slightly different from 1 in that you need to record that it was permanently deleted and that if someone posts it again, that it should be ignored, or maybe an error returned.

re: @varnerac

However, maybe the delete call is trying to "undo" the revocation by specifically deleting the version of an object that sets revocation?

You cannot undo a revocation in STIX. If it's revoked and you want the data back, you HAVE to reissue it under a new UUID. The spec is clear about this.

@jmgnc Sure. I don’t know that it prevents a STIX producer from:

1. Not publishing the version with the revocation to TAXII
2. Deleting the object, including the version of the object that held the revocation, from TAXII
3. Deleting only the object version with the revocation from the TAXII server

TAXII as a non authoritative transport can not enforce STIX rules, a specific product may do this, but there is no way for an individual TAXII server to do this. Said another way, a TAXII server can not prevent an end user from breaking STIX rules with versioning.

There are really two elements with DELETE that we need to answer (either by defining clarifying text or adding an additional endpoint or URL parameter/header option)

1. Is this just a DELETE for the content from that collection
2. Is this a DELETE of that content from every collection and STIX store (if the server has one) - I worry that this gets in to product design.

Option 2 could be handled with a universal //objects// endpoint as well.

@varnerac No, but the STIX spec says:

Implementations MUST consider the version of the STIX Object with the most recent modified value to be the most recent state of the object.

and:

A value of true in the revoked property indicates that an object (including the current version and all past versions) has been revoked.

Point 1 is irrelevant to the conversation at hand. If the revocation isn't published, it's not known to be revoked. TAXII/STIX cannot be a mind reader.

Question is, is TAXII considered an implementation of the STIX spec? And yes, I get your point about if you delete the later version, how it changes the timeline, but as the STIX spec says, once it's been revoked, there is no exception made for "deleting" the revocation object, to unrevoke the object. Notice that it specifically says that ALL past versions are considered revoked. So a conforming implementation of TAXII should, IMO, consider all older versions revoked even if the object version that says it was revoked was removed.

@jordan2175 are you saying that TAXII is not a STIX implementation? that seems a bit odd, since we expect/require that a TAXII server look at and inspect data, such at modified and other fields.

I think we are over-thinking the 'delete' operation. Delete does nothing to the object or its historical context. Delete only removes an object from the collection(s) that you request it to be deleted from. It does nothing to really delete the object in terms of existance, whether it existed or not....etc. Delete is not revocation and vis-versa.

The operation that was being requested is this:

1. Method Name: DeleteFromServerCollection
   Inputs:

• uuid of Object (mandatory);
• list of collections (optional - if empty then deleted from all collections that the client has authorized access to);
• olderThanDate (optional - if empty then delete the most recent version of the object otherwise delete all versions that are older than the specified date)

Outputs:

• Status response indicating
• number of objects deleted (mandatory)
• list of collections successfully deleted from (mandatory if collections specified)
• list of dates that matched the date filter (mandatory if olderThanDate parameter specified)

Description:
Deletes an identified object from all or set of collections that the object uuid is contained in and the client is authorized to write to that collection. If the olderThanDate is specified then all object versions of the uuid matching the specified object uuid within all specified collections will be removed.

NOTE: This does not modify the object itself at all. It only removes the object (and its versions if specified) from a collection. If a client wishes to revoke an object then the client must use be authorized to modify an object by modifying its modified date and the revoked parameter to true and then post that object to one or more collections on the TAXII server.

btw - if acceptable proposal above then I can see having a batch operation where I can specific one or more uuids to delete instead of having to call this method for each uuid. That could be a separate method that takes a list of uuids or it could just be a single method that allows one or more uuids specific as input arguments.

Do you plan to allow olderThanDate with listOfCollections in the same call? If so, the TAXII server would have to support different versions of the same object in different collections. That would probably break many existing database schemas. It breaks mine.

C1 -> O1v1, O1v2, O1v3
C2 -> O1v3

Revoked my previous comment as we actually are proposing removing a resource location so DELETE does make sense.

I think olderThanDate and listOfCollections are likely orthogonal for most cases and could support making that clarification in the text.

If that’s the case, they may be represent two distinct actions:

1. Removing an object, that is all versions of an object that share the same object id, from a collection. This is essentially “dereferencing” an object from a collection.
2. Removing underlying versions (possibly all) of an object from the underlying datastore.

Is that how you see it?

I dont think underlying data storage mechanisms is something we should focus on in the TAXII api. If something is removed from a collection then whether that object is removed in a data store on a taxii server or not is upto the TAXII server. There is nothing required to actually store persistently anything in TAXII. a TAXII server could be entirely a in-memory system that if it were to reboot it could lose all data in the collections and it would be compliant to the spec. But we still need the ability to remove things from collections.

Yeah, you’re right. Dereferencing an object from all collections is just that, and if the server wants to do something with persistence, that’s an implementation issue.

I think in general we are way over thinking this. The specification needs to address functionality that would make interactions interoperable. Providing a means for a client to tell a server to delete some content in a standardized way, seems like something we should do. Otherwise people will just add this functionality to their products in a non standardized / interoperable way.

What a server does with the request is quite frankly up to the server and its SLA with its clients/users. A server could just reject the request and say, sorry. It could say, sure I will do this and just remove it from the collection but keep it in its data store. It could print out the request and send it to someone to review and they could dispatch mice to go in to the system and eat the hard drive.

I think at a base level a DELETE at /api/collections/id/objects/id/ should remove that object from that specific collection. If you want to pass a ?match[versions]= parameter you can tell the server to match on specific versions.

We could also add a DELETE at /api/collections/id/objects/ level and allow clients to use the ?match[id]= syntax to remove a series of objects from that collection.

We could also create an endpoint of /api/objects/id/ and add a DELETE to it as a way of telling the server to delete the object from all collections. This last option could be something we do at a later time, once we figure out how badly it is needed. In the original draft of TAXII 2.0 we had this feature to provide global search. But we pulled it from the 2.0 release, so we could add it later.

Should we add the DELETE method to the /objects/ endpoint in addition to the /objects/id/ endpoint ?

The current text, heavily modified by Jason, now looks like this:

This Endpoint deletes an object from a Collection by its id. For STIX 2 objects, the {object-id} MUST be the STIX id.

If the Collection specifies can_write as false for a particular client, this Endpoint MUST return an HTTP 401 (Unauthorized), HTTP 403 (Forbidden), or HTTP 404 (Not Found) error.

If the Collection specifies can_write as true and can_read as false for a particular client, this Endpoint MUST return an HTTP 401 (Unauthorized), HTTP 403 (Forbidden), or HTTP 404 (Not Found) error. If a DELETE operation were to work on a write only collection, the result would leak information about the presence, or lack thereof, of an object in the collection.

To support removing a particular version of an object, this Endpoint supports filtering as defined in section 3.4. The only valid match parameter is version.

When a TAXII Server returns a successful response code of 200 to a DELETE, any subsequent attempts to fetch that object from the collection using the Get Objects Endpoint MUST return either an HTTP 401 (Unauthorized) or an HTTP 404 (Not Found) response. Furthermore, any subsequent attempts to fetch the collection manifest or its contents MUST NOT return the object in the result. Subsequent attempts to fetch the object from other collections present on the TAXII server MAY also return an HTTP 404 (Not Found) response, and subsequent attempts to fetch the collection manifest or its contents of other collections which contained the object MAY not return the object in the result.