Clay Icon

Trouble with write_to_table and JSON List from Claygent Webscraper

Β·
Β·

I'm having trouble getting "write_to_table" working. I've created a JSON list using the Claygent webscraper. Formatted the content properly.. but keep getting it only reading one record.

  • Avatar of Jared N.
    Jared N.
    Β·
    Β·

    Those discussions were more people with the same issue, rather than responses.

  • Avatar of Jared N.
    Jared N.
    Β·
    Β·

    Showing error that it isn't an appropriate "list".

  • Avatar of Jared N.
    Jared N.
    Β·
    Β·

    Creation of list flow.

  • Avatar of Jared N.
    Jared N.
    Β·
    Β·

    example response

  • Avatar of Jared N.
    Jared N.
    Β·
    Β·
    """{
      "company_name": [
        "Google Cloud",
        "Intel",
        "Microsoft Azure",
        "Oracle",
        "Red Hat",
        "Solo.io",
        "SUSE",
        "Akamai",
        "AWS",
        "Cast AI",
        "Datadog",
        "Dell",
        "Dockr",
        "Equinix",
        "GitHub",
        "IBM",
        "Isovalent",
        "JFrog",
        "Octopus",
        "Portworx",
        "Palo Alto Networks",
        "Splunk",
        "Sysdig",
        "Tintri",
        "VMware Tanzu",
        "Wiz",
        "Acorn",
        "Akuity",
        "Aqua Security",
        "Arm",
        "Baseten",
        "Canonical",
        "Chainguard",
        "Chronosphere",
        "Clickhouse",
        "Cloudsmith",
        "Cortex",
        "Discover",
        "Dynatrace",
        "Elastic",
        "Fastly",
        "GitLab",
        "HAProxy",
        "Harness",
        "Heroku",
        "Intuit",
        "Kong",
        "Kubecost",
        "Lambda Labs",
        "Loft",
        "MinIO",
        "New Relic",
        "Nutanix",
        "OpenSearch",
        "Rakuten Symphony",
        "ScaleOps",
        "Snyk",
        "Spectrocloud",
        "Teleport",
        "Tigera",
        "Veeam",
        "YugabyteDB",
        "Zededa",
        "AccuKnox",
        "Aembit",
        "Atolio",
        "Cerbos",
        "CHKK",
        "Cloudoor",
        "ConfidentialMind",
        "ControlTheory",
        "Cosmonic",
        "Crystal Cloud",
        "DBos",
        "DevZero",
        "DRDroid",
        "Draft.dev",
        "DragonflyDB",
        "Epsio",
        "Factory.ai",
        "Finout",
        "Flox",
        "Highlight",
        "Humanitec",
        "Infisical",
        "Jetify",
        "KTrust",
        "Kubiya.ai",
        "Kusari",
        "Massdriver",
        "Mecha",
        "Merly.ai",
        "Metoro",
        "Middleware",
        "Mirrord",
        "MoonlightMarketing",
        "N3XGEN",
        "Nethopper",
        "NudgeBee",
        "Ollama",
        "Omnistrate",
        "Oodle.ai",
        "Operant.ai",
        "Otterize",
        "P0",
        "Pelanor",
        "PerfectScale",
        "Permit.io",
        "Pipekit",
        "Quesma",
        "Rely.io",
        "Resolve.ai",
        "Restate",
        "Rig",
        "Robusta",
        "Rootly",
        "Scarf",
        "Simplyblock",
        "Speakeasy",
        "Speedscale",
        "Stateful",
        "Svix",
        "Terramate",
        "TestifySec",
        "Testkube",
        "Tracetest",
        "Wing",
        "Yovily",
        "Zenduty",
        "Zuplo",
        "Apple",
        "Lockheed Martin",
        "Robinhood"
      ],
      "company_domain": [
        "cloud.google.com",
        "intel.com",
        "azure.microsoft.com",
        "oracle.com",
        "redhat.com",
        "solo.io",
        "suse.com",
        "akamai.com",
        "aws.amazon.com",
        "cast.ai",
        "datadoghq.com",
        "dell.com",
        "dockr.ly",
        "deploy.equinix.com",
        "github.com",
        "ibm.com",
        "isovalent.com",
        "jfrog.com",
        "octopus.com",
        "portworx.com",
        "paloaltonetworks.com",
        "splunk.com",
        "sysdig.com",
        "tintri.com",
        "tanzu.vmware.com",
        "wiz.io",
        "acorn.io",
        "akuity.io",
        "aquasec.com",
        "arm.com",
        "baseten.co",
        "canonical.com",
        "chainguard.dev",
        "chronosphere.io",
        "clickhouse.com",
        "cloudsmith.com",
        "cortex.io",
        "technology.discover.com",
        "dynatrace.com",
        "elastic.co",
        "fastly.com",
        "gitlab.com",
        "haproxy.com",
        "harness.io",
        "heroku.com",
        "opensource.intuit.com",
        "konghq.com",
        "kubecost.com",
        "lambdalabs.com",
        "loft.sh",
        "min.io",
        "newrelic.com",
        "nutanix.com",
        "opensearch.org",
        "symphony.rakuten.com",
        "scaleops.com",
        "snyk.io",
        "spectrocloud.com",
        "goteleport.com",
        "tigera.io",
        "veeam.com",
        "yugabyte.com",
        "zededa.com",
        "accuknox.com",
        "aembit.io",
        "atolio.com",
        "cerbos.dev",
        "chkk.io",
        "cloudoor.com",
        "confidentialmind.com",
        "controltheory.com",
        "cosmonic.com",
        "crystal.cloud",
        "dbos.dev",
        "devzero.io",
        "drdroid.io",
        "draftt.io",
        "dragonflydb.io",
        "epsio.io",
        "factory.ai",
        "finout.io",
        "flox.dev",
        "highlight.io",
        "humanitec.com",
        "infisical.com",
        "jetify.com",
        "ktrust.io",
        "kubiya.ai",
        "kusari.dev",
        "massdriver.cloud",
        "mecha.so",
        "merly.ai",
        "metoro.io",
        "middleware.io",
        "mirrord.dev",
        "moonlightmarketing.io",
        "n3xgen.io",
        "nethopper.io",
        "nudgebee.com",
        "ollama.ai",
        "omnistrate.com",
        "oodle.ai",
        "operant.ai",
        "otterize.com",
        "p0.dev",
        "pelanor.io",
        "perfectscale.io",
        "permit.io",
        "pipekit.io",
        "quesma.com",
        "rely.io",
        "resolve.ai",
        "restate.dev",
        "rig.dev",
        "home.robusta.dev",
        "rootly.com",
        "scarf.sh",
        "simplyblock.io",
        "speakeasy.com",
        "speedscale.com",
        "stateful.com",
        "svix.com",
        "terramate.io",
        "testifysec.com",
        "testkube.io",
        "tracetest.io",
        "wing.cloud",
        "yovily.com",
        "zenduty.com",
        "zuplo.com",
        "apple.com",
        "lockheedmartin.com",
        "robinhood.com"
      ]
    }"""
  • Avatar of Jared N.
    Jared N.
    Β·
    Β·

    Followed as much as was possible in the tutorial here – but there wasn't a "structure as JSON output" option because I wasn't using openAI but rather claygent. https://www.clay.com/university/lesson/write-to-table-clay-101

  • Avatar of Fabian H.
    Fabian H.
    Β·
    Β·

    you can just create an additional GPT column and prompt it like "return me the following as JSON: " then input the claygent result and turn on the switch. Ideally you also describe the strcuture of the JSON in the prompt. This result is then a "valid list" in CLay you can use to write to another table Jared N.

  • Avatar of Fabian H.
    Fabian H.
    Β·
    Β·

    hope this helps πŸ™‚ I think I also have some video on this on LinkedIn, let me see if I can find it

  • Avatar of Bruno R.
    Bruno R.
    Β·
    Β·

    Hi Jared, thanks for reaching out! Currently, Claygent doesn’t support returning native JSON objects. However, it can return text formatted as a JSON-like structure. The good news is that our team is actively working on this feature, and we’ll share updates soon. In the meantime, I recommend using GPT-4o-mini in JSON mode, which structures the output as JSON for just 0.5 Clay credits per row. You can then utilize the "write to other table" feature to store your records in a separate table. I’ve recorded this quick video to guide you through the setup: https://www.loom.com/share/b084dbc200eb493c859ca651c5eb398e?sid=ae1a13ea-1470-451a-80db-8421b6833c16 I hope this was helpful. If there's anything else I can assist you with, please let me know!

  • Avatar of Jared N.
    Jared N.
    Β·
    Β·

    Thanks so much for the answers here. I'll take it for a spin shortly.

  • Avatar of Jared N.
    Jared N.
    Β·
    Β·

    Even in the meantime better feedback for "not a valid list" would be helpful (e.g., front-end error message)... cause I felt like I was doing everything right (but clearly wasn't). Thank you again!

  • Avatar of Channeled
    Channeled
    APP
    Β·
    Β·

    This thread was picked up by our in-app web widget and will no longer sync to Slack. If you are the original poster, you can continue this conversation by logging into https://app.clay.com and clicking "Support" in the sidebar. If you're not the original poster and require help from support, please post in 02 Support.