Streamlining Data with Functional Tables for Efficient Project Management

Jacob V. · 2024-10-10 02:18:56.871405+00

Spent some time segmenting out all my tables by function. So now I've got a ton of tables that exist to only do one thing. So now, I can use them like puzzle pieces and write data into them, and send data out of them for various projects all happening at the same time. This also makes it easier to clone a table if I need to tweak some specific functionality. Then when I make improvements on a piece of the puzzle, say a mobile phone finding workflow, those improvements apply to all projects that use that updated table! I've had it in my mind a while to systematize some things, finally getting around to it was nice.

Streamlining Data with Functional Tables for Efficient Project Management | Clay

12 comments

· Sorted by Oldest

Joe R.
·
·
Thats awesome and Ive been thinking about doing a similar thing. How are you passing data to a puzzle piece and receiving it back? Write to table + lookup? Webhook?
Joseph Z.
·
·
Love that thinking! I will definitely try that out! Thanks for sharing
Jacob V.
·
·
Joe R. yes, write to table and lookups.
Jacob V.
·
·
Just be sure to have a column in the table that is like a unique project ID. That way you pass that ID along in every single data the data moves through, so you can look up data anywhere you need to find it.
Joe R.
·
·
Thanks. I wish the lookup could be dynamic, so that when the origin value populates or updates, the lookup value will update. With the write to table > lookup approach, especially with long running processes, your lookup might occur before the final output is ready. Have you run into a similar issue?
Joe R.
·
·
If we could build an end-to-end Clay workflow and encapsulate it in a synchronous HTTP API endpoint, that would be amazing (Eric E. 😍)
Jacob V.
·
·
Joe R. I haven't had lookup timing issues, because I have conditions on all my writes and lookups. Usually I'm writing the results into a table and then after that table runs, doing the lookup last.
Jacob V.
·
·
What do you mean by wishing lookup could be dynamic? Can you tell me a bit more about this?
Joe R.
·
·
I see - yeah with write to table + lookup, one will have to think about the order of actions to make sure the lookup doesn’t run prematurely
Joe R.
·
·
Lookups are currently static, meaning the lookup happens once and the value is stored. I wish it could be dynamic, so that when the origin value updates, it will trigger an update of the downstream lookup value as well.
👍3
Dan K.
·
·
Great idea Jacob V. and had similar experience to you Joe R. and had been meaning to ask. On my journey I learnt lookup which "works" but then had to turn and burn and replace with write to table when I realised how hard to manage it would be due to being static. The project ID sounds like a good solution to reduce constant restarts in-life. Cheers!
Mathew I.
·
·
My solution for the "dynamic lookup issue" (i.e. writing data from table 1 to table 2) was using "look up row from table 1" instead of "write data to table 2" which keeps the data in the table 2 updated if data in table 1 changes. Like Jacob mentioned, you want to write a "unique id" from table 1 to table 2, and then use "look up row" in table 2 based on the unique id from table 1. That way if the data in table 1 changes, it will automatically reflect in table 2 This is opposed to writing all your data from table 1 to table 2, which is 1-time in nature and won't reflect changes.

Joe R.
·
·
Thats awesome and Ive been thinking about doing a similar thing. How are you passing data to a puzzle piece and receiving it back? Write to table + lookup? Webhook?
Joseph Z.
·
·
Love that thinking! I will definitely try that out! Thanks for sharing
Jacob V.
·
·
Joe R. yes, write to table and lookups.
Jacob V.
·
·
Just be sure to have a column in the table that is like a unique project ID. That way you pass that ID along in every single data the data moves through, so you can look up data anywhere you need to find it.
Joe R.
·
·
Thanks. I wish the lookup could be dynamic, so that when the origin value populates or updates, the lookup value will update. With the write to table > lookup approach, especially with long running processes, your lookup might occur before the final output is ready. Have you run into a similar issue?
Joe R.
·
·
If we could build an end-to-end Clay workflow and encapsulate it in a synchronous HTTP API endpoint, that would be amazing (Eric E. 😍)
Jacob V.
·
·
Joe R. I haven't had lookup timing issues, because I have conditions on all my writes and lookups. Usually I'm writing the results into a table and then after that table runs, doing the lookup last.
Jacob V.
·
·
What do you mean by wishing lookup could be dynamic? Can you tell me a bit more about this?
Joe R.
·
·
I see - yeah with write to table + lookup, one will have to think about the order of actions to make sure the lookup doesn’t run prematurely
Joe R.
·
·
Lookups are currently static, meaning the lookup happens once and the value is stored. I wish it could be dynamic, so that when the origin value updates, it will trigger an update of the downstream lookup value as well.
👍3
Dan K.
·
·
Great idea Jacob V. and had similar experience to you Joe R. and had been meaning to ask. On my journey I learnt lookup which "works" but then had to turn and burn and replace with write to table when I realised how hard to manage it would be due to being static. The project ID sounds like a good solution to reduce constant restarts in-life. Cheers!
Mathew I.
·
·
My solution for the "dynamic lookup issue" (i.e. writing data from table 1 to table 2) was using "look up row from table 1" instead of "write data to table 2" which keeps the data in the table 2 updated if data in table 1 changes. Like Jacob mentioned, you want to write a "unique id" from table 1 to table 2, and then use "look up row" in table 2 based on the unique id from table 1. That way if the data in table 1 changes, it will automatically reflect in table 2 This is opposed to writing all your data from table 1 to table 2, which is 1-time in nature and won't reflect changes.