Return All Data Lake Pipelines in One Project Deprecated

GET /api/atlas/v2/groups/{groupId}/pipelines

Returns a list of Data Lake Pipelines. To use this resource, the requesting Service Account or API Key must have the Project Read Only role.

Path parameters

  • groupId string Required

    Unique 24-hexadecimal digit string that identifies your project. Use the /groups endpoint to retrieve all projects to which the authenticated user has access.

    NOTE: Groups and projects are synonymous terms. Your group id is the same as your project id. For existing groups, your group/project id remains the same. The resource and corresponding endpoints use the term groups.

    Format should match the following pattern: ^([a-f0-9]{24})$.

Query parameters

  • envelope boolean

    Flag that indicates whether Application wraps the response in an envelope JSON object. Some API clients cannot access the HTTP response headers or status code. To remediate this, set envelope=true in the query. Endpoints that return a list of results use the results object as an envelope. Application adds the status parameter to the response body.

    Default value is false.

Responses

  • 200 application/vnd.atlas.2023-01-01+json

    OK

    Hide response attributes Show response attributes object
    • _id string

      Unique 24-hexadecimal digit string that identifies the Data Lake Pipeline.

      Format should match the following pattern: ^([a-f0-9]{24})$.

    • createdDate string(date-time)

      Timestamp that indicates when the Data Lake Pipeline was created. This parameter expresses its value in the ISO 8601 timestamp format in UTC.

    • datasetRetentionPolicy object

      Dataset Retention Policy for a Scheduled Data Lake Pipeline.

      Hide datasetRetentionPolicy attributes Show datasetRetentionPolicy attributes object
      • lastModifiedDate string(date-time)

        Date when retention policy was last modified. This parameter expresses its value in the ISO 8601 timestamp format in UTC.

      • units string Required

        Quantity of time in which the Data Lake Pipeline measures dataset retention.

        Values are DAYS, WEEKS, or MONTHS.

      • value integer(int32) Required

        Number that indicates the amount of days, weeks, or months that the Data Lake Pipeline will retain datasets.

        Minimum value is 1.

    • groupId string

      Unique 24-hexadecimal digit string that identifies the group.

      Format should match the following pattern: ^([a-f0-9]{24})$.

    • lastUpdatedDate string(date-time)

      Timestamp that indicates the last time that the Data Lake Pipeline was updated. This parameter expresses its value in the ISO 8601 timestamp format in UTC.

    • name string

      Name of this Data Lake Pipeline.

    • sink object

      Atlas Data Lake Storage as the destination for a Data Lake Pipeline.

      Hide sink attributes Show sink attributes object
      • type string Discriminator

        Type of ingestion destination of this Data Lake Pipeline.

        Value is DLS.

      • metadataProvider string

        Target cloud provider for this Data Lake Pipeline.

        Value is AWS.

      • metadataRegion string

        Target cloud provider region for this Data Lake Pipeline.

        Supported cloud provider regions
      • partitionFields array[object]

        Ordered fields used to physically organize data in the destination.

        Partition Field in the Data Lake Storage provider for a Data Lake Pipeline.

        Hide partitionFields attributes Show partitionFields attributes object
        • fieldName string Required

          Human-readable label that identifies the field name used to partition data.

          Maximum length is 700.

        • order integer(int32) Required

          Sequence in which MongoDB Cloud slices the collection data to create partitions. The resource expresses this sequence starting with zero.

          Default value is 0.

    • source object

      One of:

      On-Demand Cloud Provider Snapshots as Source for a Data Lake Pipeline.

      Hide attributes Show attributes
      • type string Discriminator

        Type of ingestion source of this Data Lake Pipeline.

        Value is ON_DEMAND_CPS.

      • clusterName string

        Human-readable name that identifies the cluster.

      • collectionName string

        Human-readable name that identifies the collection.

      • databaseName string

        Human-readable name that identifies the database.

      • groupId string

        Unique 24-hexadecimal character string that identifies the project.

        Format should match the following pattern: ^([a-f0-9]{24})$.

    • state string

      State of this Data Lake Pipeline.

      Values are ACTIVE or PAUSED.

    • transformations array[object]

      Fields to be excluded for this Data Lake Pipeline.

      Field Transformations during ingestion of a Data Lake Pipeline.

      Hide transformations attributes Show transformations attributes object
      • field string

        Key in the document.

      • type string

        Type of transformation applied during the export of the namespace in a Data Lake Pipeline.

        Value is EXCLUDE.

  • 400 application/json

    Bad Request.

    Hide response attributes Show response attributes object
    • badRequestDetail object

      Bad request detail.

      Hide badRequestDetail attribute Show badRequestDetail attribute object
      • fields array[object]

        Describes all violations in a client request.

        Hide fields attributes Show fields attributes object
        • description string Required

          A description of why the request element is bad.

        • field string Required

          A path that leads to a field in the request body.

    • detail string

      Describes the specific conditions or reasons that cause each type of error.

    • error integer(int32) Required

      HTTP status code returned with this error.

      External documentation
    • errorCode string Required

      Application error code returned with this error.

    • parameters array[object]

      Parameters used to give more information about the error.

    • reason string

      Application error message returned with this error.

  • 401 application/json

    Unauthorized.

    Hide response attributes Show response attributes object
    • badRequestDetail object

      Bad request detail.

      Hide badRequestDetail attribute Show badRequestDetail attribute object
      • fields array[object]

        Describes all violations in a client request.

        Hide fields attributes Show fields attributes object
        • description string Required

          A description of why the request element is bad.

        • field string Required

          A path that leads to a field in the request body.

    • detail string

      Describes the specific conditions or reasons that cause each type of error.

    • error integer(int32) Required

      HTTP status code returned with this error.

      External documentation
    • errorCode string Required

      Application error code returned with this error.

    • parameters array[object]

      Parameters used to give more information about the error.

    • reason string

      Application error message returned with this error.

  • 403 application/json

    Forbidden.

    Hide response attributes Show response attributes object
    • badRequestDetail object

      Bad request detail.

      Hide badRequestDetail attribute Show badRequestDetail attribute object
      • fields array[object]

        Describes all violations in a client request.

        Hide fields attributes Show fields attributes object
        • description string Required

          A description of why the request element is bad.

        • field string Required

          A path that leads to a field in the request body.

    • detail string

      Describes the specific conditions or reasons that cause each type of error.

    • error integer(int32) Required

      HTTP status code returned with this error.

      External documentation
    • errorCode string Required

      Application error code returned with this error.

    • parameters array[object]

      Parameters used to give more information about the error.

    • reason string

      Application error message returned with this error.

  • 404 application/json

    Not Found.

    Hide response attributes Show response attributes object
    • badRequestDetail object

      Bad request detail.

      Hide badRequestDetail attribute Show badRequestDetail attribute object
      • fields array[object]

        Describes all violations in a client request.

        Hide fields attributes Show fields attributes object
        • description string Required

          A description of why the request element is bad.

        • field string Required

          A path that leads to a field in the request body.

    • detail string

      Describes the specific conditions or reasons that cause each type of error.

    • error integer(int32) Required

      HTTP status code returned with this error.

      External documentation
    • errorCode string Required

      Application error code returned with this error.

    • parameters array[object]

      Parameters used to give more information about the error.

    • reason string

      Application error message returned with this error.

  • 500 application/json

    Internal Server Error.

    Hide response attributes Show response attributes object
    • badRequestDetail object

      Bad request detail.

      Hide badRequestDetail attribute Show badRequestDetail attribute object
      • fields array[object]

        Describes all violations in a client request.

        Hide fields attributes Show fields attributes object
        • description string Required

          A description of why the request element is bad.

        • field string Required

          A path that leads to a field in the request body.

    • detail string

      Describes the specific conditions or reasons that cause each type of error.

    • error integer(int32) Required

      HTTP status code returned with this error.

      External documentation
    • errorCode string Required

      Application error code returned with this error.

    • parameters array[object]

      Parameters used to give more information about the error.

    • reason string

      Application error message returned with this error.

GET /api/atlas/v2/groups/{groupId}/pipelines
atlas api listPipelines --help
import (
	"os"
	"context"
	"log"
	sdk "go.mongodb.org/atlas-sdk/v20250312001/admin"
)

func main() {
	ctx := context.Background()
	clientID := os.Getenv("MONGODB_ATLAS_CLIENT_ID")
	clientSecret := os.Getenv("MONGODB_ATLAS_CLIENT_SECRET")

	client, err := sdk.NewClient(
		sdk.UseOAuthAuth(clientID, clientSecret),
		sdk.UseBaseURL(url))

	if err != nil {
		log.Fatalf("Error: %v", err)
	}

	params = &sdk.ListPipelinesApiParams{}
	sdkResp, httpResp, err := client.DataLakePipelinesApi.
		ListPipelinesWithParams(ctx, params).
		Execute()
}
curl --header "Authorization: Bearer ${ACCESS_TOKEN}" \
  --header "Accept: application/vnd.atlas.2025-03-12+json" \
  -X GET "https://cloud.mongodb.com/api/atlas/v2/groups/{groupId}/pipelines?pretty=true"
curl --user "${PUBLIC_KEY}:${PRIVATE_KEY}" \
  --digest \
  --header "Accept: application/vnd.atlas.2025-03-12+json" \
  -X GET "https://cloud.mongodb.com/api/atlas/v2/groups/{groupId}/pipelines?pretty=true"
Response examples (200)
[
  {
    "_id": "32b6e34b3d91647abb20e7b8",
    "createdDate": "2025-05-04T09:42:00Z",
    "datasetRetentionPolicy": {
      "lastModifiedDate": "2025-05-04T09:42:00Z",
      "units": "DAYS",
      "value": 42
    },
    "groupId": "32b6e34b3d91647abb20e7b8",
    "lastUpdatedDate": "2025-05-04T09:42:00Z",
    "name": "string",
    "sink": {
      "type": "DLS",
      "metadataProvider": "AWS",
      "metadataRegion": "string",
      "partitionFields": [
        {
          "fieldName": "string",
          "order": 0
        }
      ]
    },
    "source": {
      "type": "PERIODIC_CPS",
      "clusterName": "string",
      "collectionName": "string",
      "databaseName": "string",
      "groupId": "32b6e34b3d91647abb20e7b8"
    },
    "state": "ACTIVE",
    "transformations": [
      {
        "field": "string",
        "type": "EXCLUDE"
      }
    ]
  }
]
Response examples (400)
{
  "error": 400,
  "detail": "(This is just an example, the exception may not be related to this endpoint) No provider AWS exists.",
  "reason": "Bad Request",
  "errorCode": "VALIDATION_ERROR"
}
Response examples (401)
{
  "error": 401,
  "detail": "(This is just an example, the exception may not be related to this endpoint)",
  "reason": "Unauthorized",
  "errorCode": "NOT_ORG_GROUP_CREATOR"
}
Response examples (403)
{
  "error": 403,
  "detail": "(This is just an example, the exception may not be related to this endpoint)",
  "reason": "Forbidden",
  "errorCode": "CANNOT_CHANGE_GROUP_NAME"
}
Response examples (404)
{
  "error": 404,
  "detail": "(This is just an example, the exception may not be related to this endpoint) Cannot find resource AWS",
  "reason": "Not Found",
  "errorCode": "RESOURCE_NOT_FOUND"
}
Response examples (500)
{
  "error": 500,
  "detail": "(This is just an example, the exception may not be related to this endpoint)",
  "reason": "Internal Server Error",
  "errorCode": "UNEXPECTED_ERROR"
}