Trouble with write_to_table and JSON List from Claygent Webscraper

Jared N. · 2024-10-29T17:39:29.667Z

I'm having trouble getting "write_to_table" working. I've created a JSON list using the Claygent webscraper. Formatted the content properly.. but keep getting it only reading one record.

Trouble with write_to_table and JSON List from Claygent Webscraper | Clay

13 comments

· Sorted by Oldest

Jared N.
·
·
Those discussions were more people with the same issue, rather than responses.
Jared N.
·
·
Showing error that it isn't an appropriate "list".
Jared N.
·
·
Creation of list flow.
Jared N.
·
·
example response

Jared N.

"""{
  "company_name": [
    "Google Cloud",
    "Intel",
    "Microsoft Azure",
    "Oracle",
    "Red Hat",
    "Solo.io",
    "SUSE",
    "Akamai",
    "AWS",
    "Cast AI",
    "Datadog",
    "Dell",
    "Dockr",
    "Equinix",
    "GitHub",
    "IBM",
    "Isovalent",
    "JFrog",
    "Octopus",
    "Portworx",
    "Palo Alto Networks",
    "Splunk",
    "Sysdig",
    "Tintri",
    "VMware Tanzu",
    "Wiz",
    "Acorn",
    "Akuity",
    "Aqua Security",
    "Arm",
    "Baseten",
    "Canonical",
    "Chainguard",
    "Chronosphere",
    "Clickhouse",
    "Cloudsmith",
    "Cortex",
    "Discover",
    "Dynatrace",
    "Elastic",
    "Fastly",
    "GitLab",
    "HAProxy",
    "Harness",
    "Heroku",
    "Intuit",
    "Kong",
    "Kubecost",
    "Lambda Labs",
    "Loft",
    "MinIO",
    "New Relic",
    "Nutanix",
    "OpenSearch",
    "Rakuten Symphony",
    "ScaleOps",
    "Snyk",
    "Spectrocloud",
    "Teleport",
    "Tigera",
    "Veeam",
    "YugabyteDB",
    "Zededa",
    "AccuKnox",
    "Aembit",
    "Atolio",
    "Cerbos",
    "CHKK",
    "Cloudoor",
    "ConfidentialMind",
    "ControlTheory",
    "Cosmonic",
    "Crystal Cloud",
    "DBos",
    "DevZero",
    "DRDroid",
    "Draft.dev",
    "DragonflyDB",
    "Epsio",
    "Factory.ai",
    "Finout",
    "Flox",
    "Highlight",
    "Humanitec",
    "Infisical",
    "Jetify",
    "KTrust",
    "Kubiya.ai",
    "Kusari",
    "Massdriver",
    "Mecha",
    "Merly.ai",
    "Metoro",
    "Middleware",
    "Mirrord",
    "MoonlightMarketing",
    "N3XGEN",
    "Nethopper",
    "NudgeBee",
    "Ollama",
    "Omnistrate",
    "Oodle.ai",
    "Operant.ai",
    "Otterize",
    "P0",
    "Pelanor",
    "PerfectScale",
    "Permit.io",
    "Pipekit",
    "Quesma",
    "Rely.io",
    "Resolve.ai",
    "Restate",
    "Rig",
    "Robusta",
    "Rootly",
    "Scarf",
    "Simplyblock",
    "Speakeasy",
    "Speedscale",
    "Stateful",
    "Svix",
    "Terramate",
    "TestifySec",
    "Testkube",
    "Tracetest",
    "Wing",
    "Yovily",
    "Zenduty",
    "Zuplo",
    "Apple",
    "Lockheed Martin",
    "Robinhood"
  ],
  "company_domain": [
    "cloud.google.com",
    "intel.com",
    "azure.microsoft.com",
    "oracle.com",
    "redhat.com",
    "solo.io",
    "suse.com",
    "akamai.com",
    "aws.amazon.com",
    "cast.ai",
    "datadoghq.com",
    "dell.com",
    "dockr.ly",
    "deploy.equinix.com",
    "github.com",
    "ibm.com",
    "isovalent.com",
    "jfrog.com",
    "octopus.com",
    "portworx.com",
    "paloaltonetworks.com",
    "splunk.com",
    "sysdig.com",
    "tintri.com",
    "tanzu.vmware.com",
    "wiz.io",
    "acorn.io",
    "akuity.io",
    "aquasec.com",
    "arm.com",
    "baseten.co",
    "canonical.com",
    "chainguard.dev",
    "chronosphere.io",
    "clickhouse.com",
    "cloudsmith.com",
    "cortex.io",
    "technology.discover.com",
    "dynatrace.com",
    "elastic.co",
    "fastly.com",
    "gitlab.com",
    "haproxy.com",
    "harness.io",
    "heroku.com",
    "opensource.intuit.com",
    "konghq.com",
    "kubecost.com",
    "lambdalabs.com",
    "loft.sh",
    "min.io",
    "newrelic.com",
    "nutanix.com",
    "opensearch.org",
    "symphony.rakuten.com",
    "scaleops.com",
    "snyk.io",
    "spectrocloud.com",
    "goteleport.com",
    "tigera.io",
    "veeam.com",
    "yugabyte.com",
    "zededa.com",
    "accuknox.com",
    "aembit.io",
    "atolio.com",
    "cerbos.dev",
    "chkk.io",
    "cloudoor.com",
    "confidentialmind.com",
    "controltheory.com",
    "cosmonic.com",
    "crystal.cloud",
    "dbos.dev",
    "devzero.io",
    "drdroid.io",
    "draftt.io",
    "dragonflydb.io",
    "epsio.io",
    "factory.ai",
    "finout.io",
    "flox.dev",
    "highlight.io",
    "humanitec.com",
    "infisical.com",
    "jetify.com",
    "ktrust.io",
    "kubiya.ai",
    "kusari.dev",
    "massdriver.cloud",
    "mecha.so",
    "merly.ai",
    "metoro.io",
    "middleware.io",
    "mirrord.dev",
    "moonlightmarketing.io",
    "n3xgen.io",
    "nethopper.io",
    "nudgebee.com",
    "ollama.ai",
    "omnistrate.com",
    "oodle.ai",
    "operant.ai",
    "otterize.com",
    "p0.dev",
    "pelanor.io",
    "perfectscale.io",
    "permit.io",
    "pipekit.io",
    "quesma.com",
    "rely.io",
    "resolve.ai",
    "restate.dev",
    "rig.dev",
    "home.robusta.dev",
    "rootly.com",
    "scarf.sh",
    "simplyblock.io",
    "speakeasy.com",
    "speedscale.com",
    "stateful.com",
    "svix.com",
    "terramate.io",
    "testifysec.com",
    "testkube.io",
    "tracetest.io",
    "wing.cloud",
    "yovily.com",
    "zenduty.com",
    "zuplo.com",
    "apple.com",
    "lockheedmartin.com",
    "robinhood.com"
  ]
}"""

Jared N.
·
·
Followed as much as was possible in the tutorial here – but there wasn't a "structure as JSON output" option because I wasn't using openAI but rather claygent. https://www.clay.com/university/lesson/write-to-table-clay-101
Jared N.
·
·
Here's the table: https://app.clay.com/workspaces/341873/workbooks/wb_DafunRNfTwcV/tables/t_uEfBCRymhENg/views/gv_JrBzyV3tuMs9
Fabian H.
·
·
you can just create an additional GPT column and prompt it like "return me the following as JSON: " then input the claygent result and turn on the switch. Ideally you also describe the strcuture of the JSON in the prompt. This result is then a "valid list" in CLay you can use to write to another table Jared N.
Fabian H.
·
·
hope this helps 🙂 I think I also have some video on this on LinkedIn, let me see if I can find it
Bruno R.
·
·
Hi Jared, thanks for reaching out! Currently, Claygent doesn’t support returning native JSON objects. However, it can return text formatted as a JSON-like structure. The good news is that our team is actively working on this feature, and we’ll share updates soon. In the meantime, I recommend using GPT-4o-mini in JSON mode, which structures the output as JSON for just 0.5 Clay credits per row. You can then utilize the "write to other table" feature to store your records in a separate table. I’ve recorded this quick video to guide you through the setup: https://www.loom.com/share/b084dbc200eb493c859ca651c5eb398e?sid=ae1a13ea-1470-451a-80db-8421b6833c16 I hope this was helpful. If there's anything else I can assist you with, please let me know!
Jared N.
·
·
Thanks so much for the answers here. I'll take it for a spin shortly.
Jared N.
·
·
Even in the meantime better feedback for "not a valid list" would be helpful (e.g., front-end error message)... cause I felt like I was doing everything right (but clearly wasn't). Thank you again!
Channeled
APP
·
·
This thread was picked up by our in-app web widget and will no longer sync to Slack. If you are the original poster, you can continue this conversation by logging into https://app.clay.com and clicking "Support" in the sidebar. If you're not the original poster and require help from support, please post in Clay Support.

Jared N.
·
·
Those discussions were more people with the same issue, rather than responses.
Jared N.
·
·
Showing error that it isn't an appropriate "list".
Jared N.
·
·
Creation of list flow.
Jared N.
·
·
example response

Jared N.

"""{
  "company_name": [
    "Google Cloud",
    "Intel",
    "Microsoft Azure",
    "Oracle",
    "Red Hat",
    "Solo.io",
    "SUSE",
    "Akamai",
    "AWS",
    "Cast AI",
    "Datadog",
    "Dell",
    "Dockr",
    "Equinix",
    "GitHub",
    "IBM",
    "Isovalent",
    "JFrog",
    "Octopus",
    "Portworx",
    "Palo Alto Networks",
    "Splunk",
    "Sysdig",
    "Tintri",
    "VMware Tanzu",
    "Wiz",
    "Acorn",
    "Akuity",
    "Aqua Security",
    "Arm",
    "Baseten",
    "Canonical",
    "Chainguard",
    "Chronosphere",
    "Clickhouse",
    "Cloudsmith",
    "Cortex",
    "Discover",
    "Dynatrace",
    "Elastic",
    "Fastly",
    "GitLab",
    "HAProxy",
    "Harness",
    "Heroku",
    "Intuit",
    "Kong",
    "Kubecost",
    "Lambda Labs",
    "Loft",
    "MinIO",
    "New Relic",
    "Nutanix",
    "OpenSearch",
    "Rakuten Symphony",
    "ScaleOps",
    "Snyk",
    "Spectrocloud",
    "Teleport",
    "Tigera",
    "Veeam",
    "YugabyteDB",
    "Zededa",
    "AccuKnox",
    "Aembit",
    "Atolio",
    "Cerbos",
    "CHKK",
    "Cloudoor",
    "ConfidentialMind",
    "ControlTheory",
    "Cosmonic",
    "Crystal Cloud",
    "DBos",
    "DevZero",
    "DRDroid",
    "Draft.dev",
    "DragonflyDB",
    "Epsio",
    "Factory.ai",
    "Finout",
    "Flox",
    "Highlight",
    "Humanitec",
    "Infisical",
    "Jetify",
    "KTrust",
    "Kubiya.ai",
    "Kusari",
    "Massdriver",
    "Mecha",
    "Merly.ai",
    "Metoro",
    "Middleware",
    "Mirrord",
    "MoonlightMarketing",
    "N3XGEN",
    "Nethopper",
    "NudgeBee",
    "Ollama",
    "Omnistrate",
    "Oodle.ai",
    "Operant.ai",
    "Otterize",
    "P0",
    "Pelanor",
    "PerfectScale",
    "Permit.io",
    "Pipekit",
    "Quesma",
    "Rely.io",
    "Resolve.ai",
    "Restate",
    "Rig",
    "Robusta",
    "Rootly",
    "Scarf",
    "Simplyblock",
    "Speakeasy",
    "Speedscale",
    "Stateful",
    "Svix",
    "Terramate",
    "TestifySec",
    "Testkube",
    "Tracetest",
    "Wing",
    "Yovily",
    "Zenduty",
    "Zuplo",
    "Apple",
    "Lockheed Martin",
    "Robinhood"
  ],
  "company_domain": [
    "cloud.google.com",
    "intel.com",
    "azure.microsoft.com",
    "oracle.com",
    "redhat.com",
    "solo.io",
    "suse.com",
    "akamai.com",
    "aws.amazon.com",
    "cast.ai",
    "datadoghq.com",
    "dell.com",
    "dockr.ly",
    "deploy.equinix.com",
    "github.com",
    "ibm.com",
    "isovalent.com",
    "jfrog.com",
    "octopus.com",
    "portworx.com",
    "paloaltonetworks.com",
    "splunk.com",
    "sysdig.com",
    "tintri.com",
    "tanzu.vmware.com",
    "wiz.io",
    "acorn.io",
    "akuity.io",
    "aquasec.com",
    "arm.com",
    "baseten.co",
    "canonical.com",
    "chainguard.dev",
    "chronosphere.io",
    "clickhouse.com",
    "cloudsmith.com",
    "cortex.io",
    "technology.discover.com",
    "dynatrace.com",
    "elastic.co",
    "fastly.com",
    "gitlab.com",
    "haproxy.com",
    "harness.io",
    "heroku.com",
    "opensource.intuit.com",
    "konghq.com",
    "kubecost.com",
    "lambdalabs.com",
    "loft.sh",
    "min.io",
    "newrelic.com",
    "nutanix.com",
    "opensearch.org",
    "symphony.rakuten.com",
    "scaleops.com",
    "snyk.io",
    "spectrocloud.com",
    "goteleport.com",
    "tigera.io",
    "veeam.com",
    "yugabyte.com",
    "zededa.com",
    "accuknox.com",
    "aembit.io",
    "atolio.com",
    "cerbos.dev",
    "chkk.io",
    "cloudoor.com",
    "confidentialmind.com",
    "controltheory.com",
    "cosmonic.com",
    "crystal.cloud",
    "dbos.dev",
    "devzero.io",
    "drdroid.io",
    "draftt.io",
    "dragonflydb.io",
    "epsio.io",
    "factory.ai",
    "finout.io",
    "flox.dev",
    "highlight.io",
    "humanitec.com",
    "infisical.com",
    "jetify.com",
    "ktrust.io",
    "kubiya.ai",
    "kusari.dev",
    "massdriver.cloud",
    "mecha.so",
    "merly.ai",
    "metoro.io",
    "middleware.io",
    "mirrord.dev",
    "moonlightmarketing.io",
    "n3xgen.io",
    "nethopper.io",
    "nudgebee.com",
    "ollama.ai",
    "omnistrate.com",
    "oodle.ai",
    "operant.ai",
    "otterize.com",
    "p0.dev",
    "pelanor.io",
    "perfectscale.io",
    "permit.io",
    "pipekit.io",
    "quesma.com",
    "rely.io",
    "resolve.ai",
    "restate.dev",
    "rig.dev",
    "home.robusta.dev",
    "rootly.com",
    "scarf.sh",
    "simplyblock.io",
    "speakeasy.com",
    "speedscale.com",
    "stateful.com",
    "svix.com",
    "terramate.io",
    "testifysec.com",
    "testkube.io",
    "tracetest.io",
    "wing.cloud",
    "yovily.com",
    "zenduty.com",
    "zuplo.com",
    "apple.com",
    "lockheedmartin.com",
    "robinhood.com"
  ]
}"""

Jared N.
·
·
Followed as much as was possible in the tutorial here – but there wasn't a "structure as JSON output" option because I wasn't using openAI but rather claygent. https://www.clay.com/university/lesson/write-to-table-clay-101
Jared N.
·
·
Here's the table: https://app.clay.com/workspaces/341873/workbooks/wb_DafunRNfTwcV/tables/t_uEfBCRymhENg/views/gv_JrBzyV3tuMs9
Fabian H.
·
·
you can just create an additional GPT column and prompt it like "return me the following as JSON: " then input the claygent result and turn on the switch. Ideally you also describe the strcuture of the JSON in the prompt. This result is then a "valid list" in CLay you can use to write to another table Jared N.
Fabian H.
·
·
hope this helps 🙂 I think I also have some video on this on LinkedIn, let me see if I can find it
Bruno R.
·
·
Hi Jared, thanks for reaching out! Currently, Claygent doesn’t support returning native JSON objects. However, it can return text formatted as a JSON-like structure. The good news is that our team is actively working on this feature, and we’ll share updates soon. In the meantime, I recommend using GPT-4o-mini in JSON mode, which structures the output as JSON for just 0.5 Clay credits per row. You can then utilize the "write to other table" feature to store your records in a separate table. I’ve recorded this quick video to guide you through the setup: https://www.loom.com/share/b084dbc200eb493c859ca651c5eb398e?sid=ae1a13ea-1470-451a-80db-8421b6833c16 I hope this was helpful. If there's anything else I can assist you with, please let me know!
Jared N.
·
·
Thanks so much for the answers here. I'll take it for a spin shortly.
Jared N.
·
·
Even in the meantime better feedback for "not a valid list" would be helpful (e.g., front-end error message)... cause I felt like I was doing everything right (but clearly wasn't). Thank you again!
Channeled
APP
·
·
This thread was picked up by our in-app web widget and will no longer sync to Slack. If you are the original poster, you can continue this conversation by logging into https://app.clay.com and clicking "Support" in the sidebar. If you're not the original poster and require help from support, please post in Clay Support.