Suggestions on how to upload a collection of assets that need to be in a fixed structure?

1

I am playing with creating augmented reality samples, for the Argon project, using Glitch. So far, it’s been great (esp the support turnaround to fix a few of the bugs I ran into)! One is this: Draggable Cubes

The argon.js framework (at http://argonjs.io) makes use of some parts of Cesium (http://cesiumjs.org). To do some of their operations, they need a collection of data files in a specific structure, which you point to by setting a variable:

var CESIUM_BASE_URL='https://bmaci.com/resources/cesium/';

Within that folder are a collection of directories and files that are accessed via URLs created from that variable. I’d love to be able to upload those files and keep them inside the glitch project, instead of hosting them on my personal domain (as this URL is doing).

This is a similar request to this one:

But I’m not specifically interested in just being organized, I simply can’t use these assets if they can’t be in a fixed structure. I realize it may not be feasible to support something like this, but I wanted to check.

2

Hey Blair,
the assets you upload share the same base url (eg https://cdn.glitch.com/a7b7d5c3-b931-4ce0-a746-fe06ed8da62a%2Ffrog-twitter%402x.jpg?1488681391445) that you can use a decoder like http://meyerweb.com/eric/tools/dencoder/ to output into an human readable format (eg https://cdn.glitch.com/a7b7d5c3-b931-4ce0-a746-fe06ed8da62a/[email protected]?1488681391445)

From there it’s clear to see the path that your files all use. So in my case I could do

var CESIUM_BASE_URL='https://cdn.glitch.com/a7b7d5c3-b931-4ce0-a746-fe06ed8da62a/'; .

The /a7b7d5c3-b931-4ce0-a746-fe06ed8da62a/ part is based on your project uuid so it’ll be the same for all the files you upload into your project.

3

Hey Blair,

In addition to what Pirijan already said, I’ve written a simple library: assets-lib that provides an express middleware to simulate a folder with all of your assets in it. Perhaps it would come handy for you!

4

Hi @pketh, that’s helpful but doesn’t quite solve my problem because of the additional directories. For example, if I set

var CESIUM_BASE_URL='https://cdn.glitch.com/a7b7d5c3-b931-4ce0-a746-fe06ed8da62a/'

then cesium will create URL’s of the form
https://cdn.glitch.com/a7b7d5c3-b931-4ce0-a746-fe06ed8da62a/Assets/IAU2006_XYS/IAU2000_XYS_0.json

I think the files are all in one big bin, so there is no way for me to create Assets/IAU2006_XYS/IAU2000_XYS_0.json when I upload IAU2000_XYS_0.json

5

I’m having trouble with this too. We want to upload 3D models which is a directory with 3D files (OBJ + MTL, glTF) and within those files, textures are referenced via relative paths. For example, https://github.com/KhronosGroup/glTF-Sample-Models/tree/master/1.0/VC/glTF-MaterialsCommon

Would like to be able to just upload the directory, and be able for the JS to load it out of the box with the relative paths. Otherwise, we have to point people to different upload flows (e.g., GitHub + rawgit.com), or I saw someone in the community gravitate towards neocities due to this specifically.

6

@ngokevin,
you can currently select all the files in the folder and drag them all in at once to get them all into assets. You can also set up your app so that it uses something like https://cdn.glitch.com/a7b7d5c3-b931-4ce0-a746-fe06ed8da62a (described above ) to reference relative model files).

dragging in the folder directly and it doing the right thing is something we do want to support in the future.

@blairmacintyre,
your assets are stored in a namespaced s3 bucket. The way s3 works is that it doesn’t have real folders - we simulate them with the name of the file. so folder/blah.png is basically a ‘folder’. we should be able to make relative paths in assets work by renaming an uploaded asset (eg. changing my-cool-file.jpg to folder/my-cool-file.jpg. it doesn’t yet but we’ll add that to our feature backlog.

programmatically, getting an app that’s making local/relative file system assumptions to output to a seperate CDN-ed bucket is a little trickier, but we’ll try and figure out a solution

1 Like
7

So, I tried this, and it appears that it might work already? I edited one of the asset names (didn’t even notice the name was editable before) and then copied the URL, which looked like this:

https://cdn.glitch.com/160acd26-d5a4-4a5d-a6d6-4ed87fe43bad%2Fcesium%2FAssets%2FIAU2006_XYS%2FIAU2006_XYS_0.json?1490623741029

to this

https://cdn.glitch.com/160acd26-d5a4-4a5d-a6d6-4ed87fe43bad/cesium/Assets/IAU2006_XYS/IAU2006_XYS_0.json

and pasted it my browser bar and it downloaded the file. So I’ll try

https://cdn.glitch.com/160acd26-d5a4-4a5d-a6d6-4ed87fe43bad/cesium/

as the base and see if it works!

1 Like
8

awesome, ya I suspected it might just work :smiley:

9

Well, sort of worked. Next problem. I want to get rid of the hardcoded cdn.glitch.com part, so I took the assets.js loader that @etamponi suggested and modified to support paths (e.g., but changing the top to

router.get('/*', function (request, response) {
  var name = '%2F' + encodeURIComponent(request.url.substring(1)) + '"';

it redirects appropriately (e.g., setting my path for cesium to assets/cesium and when cesium requests assets/cesium/Assets/IAU2006_XYS/IAU2006_XYS_15.json, the server will use the asset loader and issue a redirect to https://cdn.glitch.com/160acd26-d5a4-4a5d-a6d6-4ed87fe43bad%2Fcesium%2FAssets%2FIAU2006_XYS%2FIAU2006_XYS_15.json

But, now I think I’m having CORS problems again, because if I load this URL directly in my browser it works, but in my web page I get a 403 error and Preflight response is not successful

From looking around, I think it’s CORS and so I tried adding some CORS headers

  response.set("Access-Control-Allow-Origin", "*");
  response.set("Access-Control-Allow-Methods", "GET,HEAD,OPTIONS,POST,PUT");
  response.set("Access-Control-Allow-Headers", "Origin, X-Requested-With, Content-Type, Accept, Authorization");

but it doesn’t seem to help (not sure if those are really the right headers, but…)

Thoughts?

10

I’ve also tried just including the express cors package, and using it instead of setting the response headers. no luck.

Boy, it sucks to not really understand what’s going on with the control flow here! :unamused:

11

Hi @blairmacintyre,

I sure see that using the assets CDN for your use case is painful! If the images you want to use fit in the project disk, you might want to upload them there instead of using the CDN. You lose the caching layer that CloudFront provides, but perhaps it’s a better solution while we knock off these details?

12

Project disk? Not sure what you mean by that.

13

The files you see in Glitch are on a real server, that has a disk with space. Assets, on the other hand, are uploaded to a separate S3 bucket to be consumed by the CDN.

I mean that you might try to copy the files in the project, instead of uploading them in the assets CDN.

But if this makes things more complicated for you, just forget what I am saying :slight_smile: In the meantime, I am trying to see if I can make my assets lib to work with sub folders.

14

@blairmacintyre

I’ve updated the project assets-lib and it looks like it works also with subfolders now, can you give it another try?

15

I couldn’t see how to upload except to assets. If I uploaded files like this, they seemed to automatically go to assets.

The assets.js in my draggable-cubes example seems to almost work. It finds the right URL but has this error when my web page tries to access the redirect …

snt frm iOS kybrd

16

@blairmacintyre

yes, what I am saying is that I fixed the error you see in the assets.js file you have in draggable-cubes. If you can copy over the file again from the assets-lib project, you can use it without errors.

17

Note to self: don’t reply via email. @etamponi, my reply was to this message, via email:

I hadn’t yet seen

I’ll try it!

1 Like
18

Hmmm, interesting. So, in Safari it still gives the same error as before

error.png2804×148 56.2 KB

But it seems to work in Chrome and Firefox. Perhaps my attempt worked there too. Hmmm.

19

The version in http://assets-lib.glitch.me works fine for me in Safari

20

So, it worked for images at the top level: I have a crosshair_small.png that downloads fine with /assets/crosshair_small.png.

But, with the assets I renamed to have paths for names (such as cesium/Assets/IAU2006_XYS/IAU2006_XYS_15.json), it’s giving me that error.

21

I cannot reproduce, can you tell me where you are seeing this problem? I’m trying to go to that url and it works.

For example, if you visit this url http://assets-lib.glitch.me in Safari, do you see both images, or just one?

22

I’m seeing the problem in https://draggable-cubes.glitch.me

I see both of your images, and see no errors in the console. But if I look at the console with my draggable-cubes example, I see the errors.

23

I tried to load https://draggable-cubes.glitch.me with Safari, and I don’t see any error.

Are you running the latest version of Safari?

24

Define “latest”? I am still running 10.0.3, MacOS 10.12.3. I haven’t updated to any of the 10.12.4 beta’s or the 10.12.4 that was release today! Planning on updating tonight.

If you are running something more recent, that may be the cause: it appears preflight + redirects + CORS used to be disallowed, and when the spec changed, Safari was one of the last to update? https://github.com/google/shaka-player/issues/666

25

This is the version of Safari that I am using:

And this is the only error that I see in the console when I go to draggable-cubes:

Screen Shot 2017-03-27 at 23.01.55.png1271×691 80.3 KB

Perhaps you are still seeing the errors because of browser caching? Perhaps clearing the Safari cache might fix the issue for you as well.

26

Sigh. Same version of safari. I don’t see that google error, btw. If I empty (or completely disable) cache’s, nothing changes. If I disable cross-origin restrictions, it goes away (obviously).

But, if I quit safari and reload the page, I don’t get the error. Until I refresh a few times, then the error comes back. So, my guess: safari bug.

Let’s see what happens with 10.12.4 tonight.

27

Alright,

we will keep the bug open on our backlog though, to make sure we don’t forget to check in the future :slight_smile:

Thanks for your cooperation! I hope the current state works well enough for you, if you have any other problem don’t esitate to ask!

And… let’s see what happens with the new version of Safari! Keep us posted!

28

So, I think it’s working with the new Safari … but I’m not 100% sure (the joy of having the loading be deep inside someone else’s code). The previous errors go away, but I am worried that the file (the one that wasn’t downloading before) shows as unavailable in the resource browser in the debugger. That may just mean the Safari debugger can’t download it, I guess? And, of course, there are now new errors (mostly harmless, and unrelated) showing up!

Change is good, I hear. Will let you know how things pan out.

1 Like
29

TL;DR I’m having the same problems. Texture images referenced inside 3D dae files are not loaded. In other words: I’d love a user (noob) friendly way to add files, or a noob tutorial on file organizing: explaining cdn and assets, …

A year ago I created a scene in Blender for a music album, afterwards it seemed like a fun experiment to try and animate it in A-Frame. The bands site where it was hosted is offline. So I thought of redoing it on glitch, to further experiment/enhance.
I’ve got basic web knowledge. That’s what I loved about A-Frame. I’m not experienced in using JavaScript.

I’ve got dae models with textures, the textures are inside the main folder; so all I need is the cdn hosted dae files to know that the textures are in the same assets folder. Would I need to change the paths/filenames inside the dae file? Or is there a simpler way using the assets-lib referenced in this thread?

The project: electrocat.glitch.me ; for easily viewing the content of a dae file i’ve created a file there as well (it’s not used, just there for reference).

30

After working through this with the help above, I’d say the asset-lib would work. You would just need to upload all the contents of the DAE files and rename them appropriately.

To make this easier for folks in the future, it seems a solution would be:

  • (easy) include the asset-lib in the sample project (it’s small and useful) along with the modified server.js
  • (some work on glitch’s part) make it so that if I upload a directory of assets, the are named with the file paths in the names; this also simplifies uploading, and handles ambiguity if multiple files have the same names but different paths
  • (some more work on glitch’s part) the asset view really doesn’t like long names, nor does it understand “paths” in names. Fixing both (lay out the assets better so path names are legible, perhaps provide a list view in addition to the icon view, allow “directories” to be collapsed in the view so I can hide lots of assets) would make this much more usable for novices
1 Like
31

re paths in names: that’s a great suggestion. When the assets component was original written folder support from the html file api was pretty sketchy. But this may be possible now so I’ve added it to the feature backlog. Thanks :sparkles:

There may also be more we could do ui wise to help with managing assets uploaded with paths. Something we’ll look into as well.

32

Thanks for the asset-lib suggestion, and putting it in the backlog!

For beginning 3D web developers, I’m not sure how clear that model is. We’d have to start exposing backend files in our tutorials, if you export the project I think that’d mean you have to run that specific server, and uploading one-by-one is not optimal.

The best thing I’ve come up with so far is to recommend dump assets into a GitHub repo, publish to GitHub pages via the Settings, and serve via rawgit.com.

1 Like
33

I got the textures loading just fine and as expected by using the assets-lib script.
However I feel a bit out of my league as I’ve only accomplished it by some ‘blind’ copy-pasting between the example and my own electrocat A-frame project.
It’s indeed a far from intuitive experience for an intermediate 3D/Web developer.
E.g. I’ve got no prior knowledge of packages & the use of back-end / front-end folders loading automatically once using ‘package.json’ , etc…
I was just learning trough trial and error, which is fun, but was more cumbersome than just using my own server and doing some plain old copy pasting there. None the less I can imagine the benefits and ease of use for more experienced users.
I do still believe that having a solid way to structure and organising files for a project are a necessity. While beautifully chaotic and nicely illustrated, the current folder overview is not the best user experience.

So to conclude my rant onto stepping into this rabbit hole via A-Frame:

The assets-lib script readme could perhaps have a more elaborate guide with steps to perform:

  • Copy / create the necessary files:
  1. package.json (perhaps pointing to somewhere with an explanation)
  2. server.js
  3. assets.js
34

Update. I had to rewrite the assets.js script a bit to accomplish two things.

First, dealing with deleted assets. When an asset is deleted, it appears it’s line in the .glitch-assets file remains there, and a new line with it’s uuid and a deleted field is added:
{"uuid":"F43O5bp7fb4ObA7B","deleted":true}
I managed this by creating an object out of the parsed lines of the file, where each entry is a property whose name is the uuid. So, by iterating through, these deleted: true lines overwrite the previous line with the asset data. Kind of a hack, I suppose.

Second, dealing with caching in the CDN. If you delete and asset and upload a new version with the same name, they get the same URL. In the glitch UI, when you grab the URL, it puts a timestamp on the end (?37423684736) to make each version unique. So I parsed the data field into a number and added that to the URL redirect.

Here’s the updated code, if anyone wants it. I’m sure there are more elegant/compact ways of doing some of this. :slight_smile:

var express = require('express');
var fs = require('fs');

var router = express.Router();
var content = fs.readFileSync('.glitch-assets', 'utf8');
var str = content.split("\n").join(",\n");
var elems = JSON.parse('{"elems":[' + str + "{}]}").elems;
elems.pop(); // get rid of that last empty element I added to account for the final ","

var myAssets = {};
for (var val of elems) {
    myAssets[val.uuid] = val;
}

router.use((request, response) => {
  response.header("Access-Control-Allow-Origin", "*");
  response.header("Access-Control-Allow-Methods", "GET");
  response.header("Access-Control-Allow-Headers", "Origin, X-Requested-With, Content-Type, Accept, Authorization");
  console.log(request.path);
  var name = request.path.substring(1); // get rid of leading /.  used to be encodeURIComponent(request.path) + '"';
  console.log(name)
  
  for (val in myAssets) {
      if (myAssets.hasOwnProperty(val)) {
        var e = myAssets[val];
        if (e.hasOwnProperty("name") && e.name === name) {
          var time = 0;
          if (e.hasOwnProperty("date")) {
            time = Date.parse(e.date);  
          }
          var url = e.url + "?" + time;          
          console.log(url);
          return response.redirect(url);          
        }          
      }
  }

  return response.sendStatus(404);    
});

module.exports = router;
35

What would it take to get asset.js to handle space in filenames? I uploaded some pdfs to my assets folder, but only the ones without spaces are viewable by going to (xxxxx).glitch.me/assets/(xxxx).pdf

36

Are the spaces not being url encoded, so replaced with ‘%20’?

37

They are not. For example, this link works:

( since it works the forums interface automatically replaced the URL so you can’t read it, but it was my glitch domain with “/assets/Shapes.pdf” added on)

But these do not:

http://encyclopediaofcode.glitch.me/assets/Matrix Intro.pdf
http://encyclopediaofcode.glitch.me/assets/Matrix%20Intro.pdf
http://encyclopediaofcode.glitch.me/assets/Matrix-Intro.pdf

The actual URLs are, respectively,

and

38

Edited a note in, to mention that the forums automatically hides my original link

(Apologies for bumping an old topic, but this seemed to be the main place for talking about assets.js)

39

Ok, update assets.js in your project with the code at https://glitch.com/edit/#!/assets-lib?path=assets.js and files with spaces will now work

40

It works now! Thank you for this fix! (Hopefully it was a new fix and not a case of me using an old version of the file)

1 Like