So we’ve just started transitioning to using AWS S3 buckets at work (finally!). Moving away from a Windows OS fileserver that was being used to store data for people using both windows and unix ec2 instances.
Switching to s3 and moving our data in, will make it much easier for us to write code that will run on both operating systems (plus plenty of other benefits).
My colleague was getting me syntactically up to speed accessing the bucket from Python using boto3.
I found myself wanting to learn in that way that a curious child learns. Not wanting to simply remember syntax but understand context, thus making syntax easier to remember. It’s like my professors said in college: “engineers remember formulas, mathematicians remember how to derive them”, but I digress..
import boto3
s3client = boto3.client('s3')
I needed to replace this line in my code with an s3 equivalent:
for filename in os.listdir(input_dir):
Apparently with this syntax:
s3_objects = s3client.list_objects(Bucket='mybucket')
me: “So, what’s an object in this context?” I asked.
e: “It’s anything in an s3 bucket”.
Fair enough. That gets my what I want. But wait…
me: “What about sub directories, how does it handle those?”
e: “like this”:
s3_objects = s3client.list_objects(Bucket='mybucket', Prefix='subdir/')
me: “ok, wait, why is it called Prefix? Isn’t it one level lower down?”
e: “Yes and No. There aren’t actually any subdirectories in an s3 bucket. Just, files in this bucket. This is a prefix on the filename”: subdir/somedata.csv
me: “ah ok, I get it. Wait.. how is that not a subdirectory?..”
This got me thinking. The question of whether this is a sub directory or not is really just a question of how the api works, it has nothing to do with the absoluteness of how the data is stored. We’re only ever one abstraction away from an entirely different perception. I have no idea what the code looks like underneath the creation of subdirectories in an operating system. Of course there are all kinds of important CS decisions made for optimization, et cetera, underneath the hood. But ultimately- there is some kind of user experience on top. We not looking at 1s and 0s.
Thru abstraction we can create any appearance we want
So is this the case with perception in general?.. and what if we didn’t know there was an abstraction on top? Or is there always an abstraction?
So what is the point of an abstraction. Well it can have a few different purposes such as:
- Creating a fixed interface that allows users to work with it. Then if something needs to be changed under the hood, it can be, without affecting the user experience.
- Taking a more low level interface to higher level interface. i.e. More Complexity, less functionality, easier use, but if something goes wrong, its hard to know what to do.
As such, in terms of point 2. I think this abstraction is Analogous with the concept of Holons. I.e. Such abstraction seems to occur naturally in life. Atoms make up molecules make up organs in our body make up a whole human body. So are we always “looking” at some kind of abstraction? Then what is absolute? Does it matter? It’s like Morpheus said:
Lets suppose then for a second that the universe is a simulation.
Just for fun then lets ponder if maybe then point 1. Could hold as well! Maybe the abstraction, UX, could remain fixed while changes are made at a lower level (pre sub-atomic particles) that we’re unable to perceive.
I have friends who conceptualize the underlying process of the universe from a purely “rational”, and reductionist perspective and others that describe it all in terms of God. A very loaded word with many different interpretations from a man in the clouds to the actual unfolding process of the universe itself.
To me though, debating such things, fighting over such things, killing each other over such things, seems like debating whether or not the Prefix in an s3 bucket is a sub directory or not. A perception on top of an unseen underlying process. It’s like Macklemore said
Whatever God you believe in,
We come from the same one
and you know what else he said:
Strip away the fear
Underneath it’s all the same love
Maybe, there is another perception available inside us that allows us to connect to something pre-everything. To read the source code, with clarity and understanding.
Robert David West 