[Resolved] Service Incident Update: Projects Loading Slowly or Not At All, NPM Reinstalls, & Missing Code

Glitch Service Updates
1

Ongoing Incident Update - We are still working to bring the site fully back up.

Once this incident is marked “Resolved” on status.glitch.com, most of these issues will fix themselves.

  • loading slowly or not at all
  • not responding
  • github import/export errors
  • npm reinstalls

Folks with the following issues will need to send a message to [email protected] so we can help you out:

  • suspension for preparing error
  • suspension for disk space
  • missing code - if you are missing code, don’t open the project editor further so we can restore your project to a backup.
4 Likes
3

Yesterday, I had put multiple hours of time into one of my glitch projects, to log in today and find all of it gone, completely unsaved despite it being saved properly yesterday. I checked rewind to see if something happened, and it appears that everything that happened yesterday is no longer in rewind except a single change that happened at the start of the day.

Is there anything I can do to recover the changes made yesterday? and what happened to all of the stuff that was saved properly yesterday.

1 Like
4

I’ve had this issue as well! I noticed that it happened after I remixed the project, which was helpful because the remix served as a makeshift backup, though it was tragic for me to experience and I’m now paranoid that none of my progress will actually be saved as well.

5

It’s very severe, it will probably take hours to reimplement everything that is lost. We did remixes of the project earlier but none of them yesterday, and there seems to be no backups or anything. I’ve seen glitch having a lot of connection problems throughout the last few days but this is the worst thing that has happened yet.

6

Try contacting : [email protected]
Do no make any modification as it might erase previous version.

1 Like
7

I’ve already started rewriting a bunch of it, and I don’t really need to get it restored, but it’s still very upsetting that glitch did this, and my fear is that it may happen again. Going through support and not modifying the project at all will probably take longer then just rewriting the deleted code (and I’ve also already modified the project).

8

This happened to me a long time ago and I got it back within 24 hours, but this was weeks of work. It seemed to get reverted as if someone at Glitch Support reverted it back to a previous backup (e.g. Rewind was reverted as well) without me asking (they didn’t). Anyway they can get it back, but it seems this is a returning issue for Glitch users so maybe @glitch_support could look into it :slight_smile:

9

Thanks for letting us know!

10

When I ran enable-pnpm I got this error:
ERROR ENOSPC: no space left on device, open ‘/rbd/pnpm-volume/store/2/store.json.1783575922’
I checked and im no where near 400mb. Nothing works now since I keep getting a can’t find module error.

Edit: This is where it all begins… (cus it was moved lmao)

2 Likes
11

@glitch_support please help I have no idea why this is happening

12

Use git gc or git prune to get a little bit of a space.

1 Like
13

Already tried that and it did nothing

14

Try with rm -rf /app/.git/. It should works.

15

After running that and doing git gc this happened so it didn’t fix :frowning:

image
image750×206 57.9 KB

16

No space left :joy:

image
image750×234 34.4 KB

17

There is space. SPACE!!!

18

what… :no_mouth:

19

Good call posting it here. /rbd/pnpm-volume/store/2/store.json.1783575922 is in the bare container disk, i.e. not in the project’s /app “400 MB limit” drive, and not in the node_modules drive either, and not even on that /tmp mount from outside. It makes sense that clearing out the /app/.git dir, which lives in the project drive, didn’t fix it. Sorry about your project’s commit history, btw.

But just for the sake of having a written record, could you post the output of this:

df -h / /tmp /app

thanks

If there’s anything big in your container’s /tmp, maybe try erasing those \:
… but if this server is really under a lot of disk pressure, you might not even get that space back ):

20

Gimme a second I’m on mobile

21

Did you try with df -h in your terminal?

22

image
image750×441 96.6 KB

This is the result @wh0

1 Like
23

image
image750×378 62.7 KB

(This one has a line that’s missing in the first one)

24

Did you try with git prune or removing useless packages?

25

I’m pretty sure I’m using all my packages

26

Omg I think git prune fixed it

27

I’ll post something when it’s done

28

Okay, i got it successfully.

29

This happened

image
image750×460 115 KB
image
image750×448 149 KB

30

And I’m getting the same can’t find module errors again. What should I do??

31

this kind of looks like pnpm isn’t set up right. it’s not supposed to put a registry in /rbd/pnpm-volume/store/2. it’s supposed to connect to a server through a socket in that directory, I think. I’m a beginner at this stuff, but that’s what my container has, anyway.

Could you show the output of this:

pnpm server status

On mine the output is

store: /rbd/pnpm-volume/store/2
process id: 31766
remote prefix: http://unix:/rbd/pnpm-volume/store/2/server/socket:

edit: oh neat, the pnpm author built the store system for Glitch https://dev.to/zkochan/comment/416g

32

image
image750×217 60.6 KB
@wh0

1 Like
33

Weird, i use npm, not pnpm. With using npm, this issue isn’t occurring in any way.

35

As i mentioned, in my case i only used git prune && git gc and it fixed my issue.

36

So does anyone else have a solution…

37

Contact Glitch support ([email protected]) and they can likely give some more advice to help you work it out. Really sorry that this is happening!

38

Ok I’ll try to contact them

1 Like
39

when I ran pnpm i I got this:
ENOSPC: no space left on device, mkdir ‘/rbd/pnpm-volume/93b21c41-037b-43a7-a1b3-7415a059ac51/node_modules/.registry.npmjs.org/better-sqlite3/6.0.1’

what does better-sqlite3 have to do with this!?!?!?

40

There is a issue with Glitch’s end. The same issue happening to me even i deleted the rewind cache. @glitch_support Can you response to this message, or listening to the community and fixing the current issues.

1 Like
41

Thanks for getting back to us with an update. Again, there’s gonna be no solutions here, because this stuff is out of our control. But this one is interesting. This path makes sense: it’s where the modules get “copied” to by pnpm. This area appears to be on a btrfs filesystem, which makes sense, because it has neat features that would allow you to copy these npm packages around without using duplicate disk space. But it also has this additional complexity that even if you’re not out of space in terms of bytes available, you can run out of “metadata chunks.”

Running out of metadata chunks–seems like it could happen if the disk were storing a bunch of node modules, which tend to have many small files.

According to this thing on stackexchange, there’s a command one could run to “balance” the free space with free metadata chunks. But unfortunately we’re not in charge of that disk as a whole, and we users can’t do anything about this.

2 Likes
42

should I try remixing :flushed:

43

If I were desperate, I probably would. There’s a chance that it would land on another host that’s not experiencing this.

44

when do you think this would be fixed (like a rough estimate)

45

I broke glitch in that point, so my project is now having root permissions. WTF

46

noice lol. glitch is really broken these days.

47

I have the same problem, I tried git prune && git gc but nothing

48

Looks like this is a Glitch/site-wide issue, not specific to projects. For more info on the outage, see: status.glitch.com. You may want to email [email protected] or create a ticket on Happyfox

3 Likes
49

Oh I didn’t think this would be part of the outage.

50

Well, I have seen multiple people with the same issue and they are experiencing bursts of downtime and outages, so I think it might be linked somehow, or be something that Glitch need to fix on their end

2 Likes
51

there’s nothing wrong with that, you have your own container, you have SU permissions

52

I seemed to be able to resolve this issue by running enable-pnpm followed by enable-npm

1 Like
53

Having the same issue and I’ve tried every solution in this thread, but none of them works.
Here’s also me doing df -h / /tmp /app
image

1 Like
54

Wow, 100K free is not gonna be enough to install dependencies. Thanks for the screenshot.

1 Like
55

Screen Shot 2020-06-21 at 1.39.23 PM
@wh0 is this enough?? lol

56

yeah. might be worth trying npm instead of pnpm, with all that space you @Daniel have in /app

57

OMFG I TYPED enable-npm AND NOW I CANT REWIND AND MY SOME CODE GOT REPLACED WITH BLACK DOTS!!! @wh0 DO YOU KNOW HOW TO FIX THIS!!!

58

oh geez, this?

59

Yes it got replaced with those things. PLEASE HELP!! IM ******* PANICKING

60

and you said earlier that you deleted the .git dir? do you have any backups?

61

I have one back up on my computer from June 14

62

Ok I just checked and it’s only some files so hopefully I’m not totally ******.

63

4 files are just replaced with dots :cry: good thing I backed up my project. I think my .sqlite files are fine and I should be able to paste in the code of those 4 files since I haven’t changed them since June 14.

1 Like
64

Code pasted! Everything is working now!!

65

@wh0 im getting this > UnhandledPromiseRejectionWarning: SqliteError: disk I/O error
now. Do you know to fix it??

66

I’ve been seeing this intermittently. That is to say, after many subsequent attempts, a later disk write would succeed.

image

Above: a cluster of I/O errors observed last night. red-orange = I/O error, light blue = success. time on vertical axis, more recent entries on top, ~1 second per pixel. longer bars mean longer write latency.

But I wasn’t using sqlite in that case. I don’t know if the error would corrupt the database. (In my case, I was rewriting the entire file.) It could even be that these I/O errors are what cause files to become replaced with dots, e.g. when the Glitch editor encounters such an error.

So at some level, maybe the solution is to do your best to back up the database and try to rebuild your app to store its state on a third party service.

Or if you want to try something wild, just retry the write until it succeeds. Again, I don’t know if an error would corrupt the database.

67

I have the same issue :

node v12.0.0, with pnpm

9:01 PM

Installing…

9:01 PM

Performing headless installation

9:01 PM

ERROR ENOSPC: no space left on device, open ‘/rbd/pnpm-volume/store/2/store.json.2640651246’

1 Like
68

@wh0 should I just try uploading the old sqlite file from my backup?? Don’t really care about the data since it isn’t important anyway. Do you think it’ll work?

69

Welp, I’ve got the exact same issue with pnpm…

70

could it be cus of the outage?!?!?

71

It no doubt is caused by the outage, so we’ll have to wait for Glitch to fix this…

72

Wow, that’s a good situation to be in. It’ll save you the trouble of figuring out the details of how sqlite works. If you can upload the backup version of the database–and it may take multiple tries if there are intermittent errors–then you can be sure (more sure anyway :roll_eyes: so much uncertainty right now) that whatever’s saved is not corrupted. At least not corrupted until the next write attempt.

73

When I type enable-pnpm or enable-npm I have this :

rm: cannot remove ‘node_modules’: Structure needs cleaning
restarting…

74

ok ill try it soon!!

75

@wh0 should I download my current project just in case??

76

If we were to choose between backups or no backups, with all else being equal, we’d choose backups every time. What would the downsides even be?

1 Like
77

wait how do I upload files to glitch again :flushed: I pressed upload file and selected the file and it uploaded but how do I get it in my project?

78

wget <glitch cdn url>

79

the file is empty :flushed: :flushed:

80

Are referring to that thing where you upload an “asset” to a CDN?

And you uploaded it and the CDN gives back an empty file? You might be the first user to report that.

81

ok it failed the first time but I tried it again and now it works!!

82

lets see if this fixed the issue!

83

my project is taking a while to load :cry:

84

Here’s my logs after doing enable-pnpm after a lot of times sending that command without success :

node v12.0.0, with pnpm

9:51 PM

Installing...

9:51 PM

Performing headless installation

9:51 PM

Resolving: total 1, reused 0, downloaded 0

9:52 PM

Resolving: total 1, reused 0, downloaded 0

9:52 PM

Resolving: total 1, reused 1, downloaded 0

9:52 PM

Resolving: total 1, reused 1, downloaded 0

9:52 PM

Resolving: total 1, reused 1, downloaded 0

9:52 PM

Resolving: total 1, reused 1, downloaded 0

9:52 PM

Resolving: total 1, reused 1, downloaded 0

9:52 PM

Resolving: total 1, reused 1, downloaded 0

9:52 PM

Resolving: total 1, reused 1, downloaded 0

9:52 PM

Resolving: total 2, reused 1, downloaded 0

9:52 PM

Resolving: total 3, reused 3, downloaded 0

9:52 PM

Resolving: total 3, reused 3, downloaded 0

9:52 PM

Resolving: total 3, reused 3, downloaded 0

9:52 PM

Resolving: total 3, reused 3, downloaded 0

9:52 PM

Resolving: total 4, reused 3, downloaded 0

9:52 PM

Resolving: total 7, reused 7, downloaded 0

9:52 PM

Resolving: total 7, reused 7, downloaded 0

9:52 PM

Resolving: total 7, reused 7, downloaded 0

9:52 PM

Resolving: total 7, reused 7, downloaded 0

9:52 PM

Resolving: total 7, reused 7, downloaded 0

9:52 PM

Resolving: total 7, reused 7, downloaded 0

9:52 PM

Resolving: total 7, reused 7, downloaded 0

9:52 PM

Resolving: total 7, reused 7, downloaded 0

9:52 PM

Resolving: total 7, reused 7, downloaded 0

9:52 PM

Resolving: total 7, reused 7, downloaded 0

9:52 PM

Resolving: total 7, reused 7, downloaded 0

9:52 PM

Resolving: total 7, reused 7, downloaded 0

9:52 PM

Resolving: total 7, reused 7, downloaded 0

9:52 PM

Resolving: total 8, reused 7, downloaded 0

9:52 PM

Resolving: total 9, reused 8, downloaded 0

9:52 PM

Resolving: total 10, reused 9, downloaded 0

9:52 PM

Resolving: total 10, reused 9, downloaded 0

9:52 PM

Resolving: total 10, reused 9, downloaded 0

9:52 PM

Resolving: total 10, reused 9, downloaded 0

9:52 PM

Resolving: total 10, reused 9, downloaded 0

9:52 PM

Resolving: total 10, reused 9, downloaded 0

9:52 PM

Resolving: total 11, reused 9, downloaded 0

9:52 PM

Resolving: total 14, reused 12, downloaded 0

9:52 PM

Resolving: total 35, reused 34, downloaded 0

9:52 PM

Resolving: total 44, reused 43, downloaded 0

9:52 PM

Resolving: total 49, reused 47, downloaded 0

9:52 PM

Resolving: total 49, reused 47, downloaded 0

9:52 PM

Resolving: total 49, reused 47, downloaded 0

9:52 PM

Resolving: total 53, reused 53, downloaded 0

9:52 PM

Resolving: total 65, reused 63, downloaded 0

9:52 PM

Resolving: total 83, reused 82, downloaded 0

9:52 PM

Resolving: total 104, reused 103, downloaded 0

9:52 PM

Resolving: total 119, reused 119, downloaded 0

9:52 PM

Resolving: total 144, reused 143, downloaded 0

9:52 PM

Resolving: total 160, reused 159, downloaded 0

9:52 PM

Resolving: total 168, reused 167, downloaded 0

9:52 PM

Resolving: total 181, reused 180, downloaded 0

9:52 PM

Resolving: total 193, reused 190, downloaded 0

9:52 PM

Resolving: total 205, reused 203, downloaded 1

9:52 PM

Resolving: total 214, reused 211, downloaded 1

9:52 PM

Resolving: total 222, reused 219, downloaded 1

9:52 PM

Resolving: total 223, reused 219, downloaded 1

9:52 PM

Resolving: total 225, reused 221, downloaded 1

9:52 PM

Resolving: total 226, reused 223, downloaded 1

9:52 PM

Resolving: total 229, reused 227, downloaded 1

9:53 PM

Resolving: total 229, reused 227, downloaded 1

9:53 PM

Resolving: total 229, reused 227, downloaded 1

9:53 PM

Resolving: total 230, reused 228, downloaded 1

9:53 PM

Resolving: total 231, reused 229, downloaded 1

9:53 PM

Resolving: total 231, reused 229, downloaded 1

9:53 PM

Resolving: total 232, reused 230, downloaded 1

9:53 PM

Resolving: total 232, reused 230, downloaded 1

9:53 PM

Resolving: total 233, reused 231, downloaded 1

9:53 PM

Resolving: total 235, reused 231, downloaded 1

9:53 PM

Resolving: total 235, reused 231, downloaded 1

9:53 PM

Resolving: total 235, reused 231, downloaded 1

9:53 PM

Resolving: total 235, reused 231, downloaded 1

9:53 PM

Resolving: total 236, reused 231, downloaded 1

9:53 PM

Resolving: total 236, reused 231, downloaded 1

9:53 PM

Resolving: total 237, reused 231, downloaded 1

9:53 PM

Resolving: total 237, reused 231, downloaded 1

9:53 PM

Resolving: total 238, reused 231, downloaded 1

9:53 PM

Resolving: total 238, reused 231, downloaded 1

9:53 PM

Resolving: total 238, reused 231, downloaded 1

9:53 PM

Resolving: total 240, reused 232, downloaded 1

9:53 PM

Resolving: total 240, reused 232, downloaded 1

9:53 PM

Resolving: total 241, reused 233, downloaded 1

9:53 PM

Resolving: total 241, reused 233, downloaded 1

9:53 PM

Resolving: total 241, reused 233, downloaded 1

9:53 PM

Resolving: total 242, reused 233, downloaded 1

9:53 PM

Resolving: total 242, reused 234, downloaded 1

9:53 PM

Resolving: total 242, reused 234, downloaded 1

9:53 PM

Resolving: total 243, reused 234, downloaded 1

9:53 PM

Resolving: total 244, reused 234, downloaded 1

9:53 PM

Resolving: total 244, reused 234, downloaded 1

9:53 PM

Resolving: total 245, reused 236, downloaded 1

9:53 PM

Resolving: total 245, reused 236, downloaded 1

9:53 PM

Resolving: total 245, reused 236, downloaded 1

9:53 PM

Resolving: total 246, reused 237, downloaded 1

9:53 PM

Resolving: total 246, reused 237, downloaded 1

9:53 PM

Resolving: total 247, reused 237, downloaded 1

9:53 PM

Resolving: total 247, reused 237, downloaded 1

9:53 PM

Resolving: total 247, reused 238, downloaded 1

9:53 PM

Resolving: total 247, reused 238, downloaded 1

9:53 PM

Resolving: total 247, reused 238, downloaded 1

9:53 PM

Resolving: total 247, reused 238, downloaded 1

9:53 PM

Resolving: total 247, reused 238, downloaded 1

9:53 PM

Resolving: total 248, reused 238, downloaded 1

9:53 PM

Resolving: total 248, reused 238, downloaded 1

9:53 PM

Resolving: total 249, reused 238, downloaded 1

9:53 PM

Resolving: total 250, reused 240, downloaded 1

9:53 PM

Resolving: total 250, reused 240, downloaded 1

9:53 PM

Resolving: total 250, reused 240, downloaded 1

9:53 PM

Resolving: total 250, reused 240, downloaded 1

9:53 PM

Resolving: total 252, reused 241, downloaded 1

9:53 PM

Resolving: total 253, reused 243, downloaded 1

9:53 PM

Resolving: total 253, reused 243, downloaded 1

9:53 PM

Resolving: total 253, reused 243, downloaded 1

9:53 PM

Resolving: total 254, reused 243, downloaded 1

9:53 PM

Resolving: total 254, reused 243, downloaded 1

9:53 PM

Resolving: total 254, reused 243, downloaded 1

9:53 PM

Resolving: total 255, reused 244, downloaded 1

9:53 PM

Resolving: total 255, reused 244, downloaded 1

9:53 PM

Resolving: total 255, reused 244, downloaded 1

9:53 PM

Resolving: total 255, reused 244, downloaded 1

9:54 PM

Resolving: total 256, reused 245, downloaded 1

9:54 PM

Resolving: total 256, reused 245, downloaded 1

9:54 PM

Resolving: total 256, reused 245, downloaded 1

9:54 PM

Resolving: total 257, reused 246, downloaded 1

9:54 PM

Resolving: total 257, reused 246, downloaded 1

9:54 PM

Resolving: total 257, reused 246, downloaded 1

9:54 PM

Resolving: total 257, reused 246, downloaded 1

9:54 PM

Resolving: total 258, reused 246, downloaded 1

9:54 PM

Resolving: total 258, reused 246, downloaded 1

9:54 PM

Resolving: total 258, reused 246, downloaded 1

9:54 PM

Resolving: total 258, reused 246, downloaded 1

9:54 PM

Resolving: total 258, reused 246, downloaded 1

9:54 PM

Resolving: total 258, reused 246, downloaded 1

9:54 PM

Resolving: total 259, reused 248, downloaded 1

9:54 PM

Resolving: total 260, reused 249, downloaded 1

9:54 PM

Resolving: total 260, reused 249, downloaded 1

9:54 PM

Resolving: total 260, reused 249, downloaded 1

9:54 PM

Resolving: total 260, reused 249, downloaded 1

9:54 PM

Resolving: total 262, reused 251, downloaded 1

9:54 PM

Resolving: total 263, reused 252, downloaded 1

9:54 PM

Resolving: total 263, reused 252, downloaded 1

9:54 PM

Resolving: total 263, reused 253, downloaded 1

9:54 PM

Resolving: total 264, reused 253, downloaded 1

9:54 PM

Resolving: total 264, reused 253, downloaded 1

9:54 PM

Resolving: total 265, reused 254, downloaded 1

9:54 PM

Resolving: total 265, reused 255, downloaded 1

9:54 PM

Resolving: total 265, reused 255, downloaded 1

9:54 PM

Resolving: total 266, reused 255, downloaded 1

9:54 PM

Resolving: total 266, reused 255, downloaded 1

9:54 PM

Resolving: total 268, reused 258, downloaded 1

9:54 PM

Resolving: total 268, reused 258, downloaded 1

9:54 PM

Resolving: total 268, reused 258, downloaded 1

9:54 PM

Resolving: total 269, reused 258, downloaded 1

9:54 PM

Resolving: total 270, reused 262, downloaded 1

9:54 PM

Resolving: total 271, reused 263, downloaded 1

9:54 PM

Resolving: total 272, reused 264, downloaded 1

9:54 PM

Resolving: total 272, reused 264, downloaded 1

9:54 PM

Resolving: total 272, reused 264, downloaded 1

9:54 PM

Resolving: total 273, reused 264, downloaded 1

9:54 PM

Resolving: total 273, reused 264, downloaded 1

9:54 PM

Resolving: total 274, reused 265, downloaded 1

9:54 PM

Resolving: total 274, reused 265, downloaded 1

9:54 PM

Resolving: total 274, reused 265, downloaded 1

9:54 PM

Resolving: total 275, reused 265, downloaded 1

9:54 PM

Resolving: total 275, reused 265, downloaded 1

9:54 PM

Resolving: total 276, reused 267, downloaded 1

9:54 PM

Resolving: total 276, reused 267, downloaded 1

9:54 PM

Resolving: total 277, reused 268, downloaded 1

9:54 PM

Resolving: total 279, reused 271, downloaded 1

9:54 PM

Resolving: total 279, reused 271, downloaded 1

9:54 PM

Resolving: total 279, reused 271, downloaded 1

9:54 PM

Resolving: total 280, reused 272, downloaded 1

9:54 PM

Resolving: total 280, reused 272, downloaded 1

9:54 PM

Resolving: total 289, reused 285, downloaded 1

9:54 PM

Resolving: total 290, reused 286, downloaded 1

9:54 PM

Resolving: total 293, reused 289, downloaded 1

9:54 PM

Resolving: total 297, reused 293, downloaded 1

9:54 PM

Resolving: total 306, reused 297, downloaded 1

9:54 PM

Resolving: total 311, reused 303, downloaded 1

9:54 PM

Resolving: total 311, reused 303, downloaded 1

9:55 PM

Resolving: total 312, reused 303, downloaded 1

9:55 PM

Resolving: total 312, reused 303, downloaded 1

9:55 PM

Resolving: total 312, reused 303, downloaded 1

9:55 PM

Resolving: total 313, reused 303, downloaded 1

9:55 PM

Resolving: total 314, reused 305, downloaded 1

9:55 PM

Resolving: total 318, reused 307, downloaded 1

9:55 PM

Resolving: total 325, reused 317, downloaded 1

9:55 PM

Resolving: total 332, reused 325, downloaded 1

9:55 PM

Resolving: total 338, reused 331, downloaded 1

9:55 PM

Packages: +338

9:55 PM

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

9:55 PM

 ERROR  ENOSPC: no space left on device, mkdir '/rbd/pnpm-volume/store/2/registry.npmjs.org/twitch-webhooks'

9:55 PM

Error: Promise rejected with value: {

9:55 PM

errno: -28,

9:55 PM

code: 'ENOSPC',

9:55 PM

syscall: 'mkdir',

9:55 PM

path: '/rbd/pnpm-volume/store/2/registry.npmjs.org/twitch-webhooks',

9:55 PM

message: 'ENOSPC: no space left on device, mkdir ' +

9:55 PM

"'/rbd/pnpm-volume/store/2/registry.npmjs.org/twitch-webhooks'"

9:55 PM

}

9:55 PM

at /opt/nvm/versions/node/v10.15.3/lib/node_modules/pnpm/lib/node_modules/loud-rejection/index.js:27:12

9:55 PM

at Array.forEach (<anonymous>)

9:55 PM

at EventEmitter.<anonymous> (/opt/nvm/versions/node/v10.15.3/lib/node_modules/pnpm/lib/node_modules/loud-rejection/index.js:23:24)

9:55 PM

at EventEmitter.emit (events.js:196:13)

9:55 PM

at emit (/opt/nvm/versions/node/v10.15.3/lib/node_modules/pnpm/lib/node_modules/signal-exit/index.js:77:11)

9:55 PM

at processEmit [as emit] (/opt/nvm/versions/node/v10.15.3/lib/node_modules/pnpm/lib/node_modules/signal-exit/index.js:150:5)

9:55 PM

at process.exit (internal/process/per_thread.js:157:15)

9:55 PM

at Timeout._onTimeout (/opt/nvm/versions/node/v10.15.3/lib/node_modules/pnpm/lib/err.js:10:30)

9:55 PM

at listOnTimeout (internal/timers.js:531:17)

9:55 PM

at processTimers (internal/timers.js:475:7)

9:55 PM

Error: Promise rejected with value: {

9:55 PM

errno: -28,

9:55 PM

code: 'ENOSPC',

9:55 PM

syscall: 'mkdir',

9:55 PM

path: '/rbd/pnpm-volume/store/2/registry.npmjs.org/twitch-webhooks',

9:55 PM

message: 'ENOSPC: no space left on device, mkdir ' +

9:55 PM

"'/rbd/pnpm-volume/store/2/registry.npmjs.org/twitch-webhooks'"

9:55 PM

}

9:55 PM

at /opt/nvm/versions/node/v10.15.3/lib/node_modules/pnpm/lib/node_modules/loud-rejection/index.js:27:12

9:55 PM

at Array.forEach (<anonymous>)

9:55 PM

at EventEmitter.<anonymous> (/opt/nvm/versions/node/v10.15.3/lib/node_modules/pnpm/lib/node_modules/loud-rejection/index.js:23:24)

9:55 PM

at EventEmitter.emit (events.js:196:13)

9:55 PM

at emit (/opt/nvm/versions/node/v10.15.3/lib/node_modules/pnpm/lib/node_modules/signal-exit/index.js:77:11)

9:55 PM

at processEmit [as emit] (/opt/nvm/versions/node/v10.15.3/lib/node_modules/pnpm/lib/node_modules/signal-exit/index.js:150:5)

9:55 PM

at process.exit (internal/process/per_thread.js:157:15)

9:55 PM

at Timeout._onTimeout (/opt/nvm/versions/node/v10.15.3/lib/node_modules/pnpm/lib/err.js:10:30)

9:55 PM

at listOnTimeout (internal/timers.js:531:17)

9:55 PM

at processTimers (internal/timers.js:475:7)

9:55 PM

Error: Promise rejected with value: {

9:55 PM

errno: -28,

9:55 PM

code: 'ENOSPC',

9:55 PM

syscall: 'mkdir',

9:55 PM

path: '/rbd/pnpm-volume/store/2/registry.npmjs.org/@types/public-ip',

9:55 PM

message: 'ENOSPC: no space left on device, mkdir ' +

9:55 PM

"'/rbd/pnpm-volume/store/2/registry.npmjs.org/@types/public-ip'"

9:55 PM

}

9:55 PM

at /opt/nvm/versions/node/v10.15.3/lib/node_modules/pnpm/lib/node_modules/loud-rejection/index.js:27:12

9:55 PM

at Array.forEach (<anonymous>)

9:55 PM

at EventEmitter.<anonymous> (/opt/nvm/versions/node/v10.15.3/lib/node_modules/pnpm/lib/node_modules/loud-rejection/index.js:23:24)

9:55 PM

at EventEmitter.emit (events.js:196:13)

9:55 PM

at emit (/opt/nvm/versions/node/v10.15.3/lib/node_modules/pnpm/lib/node_modules/signal-exit/index.js:77:11)

9:55 PM

at processEmit [as emit] (/opt/nvm/versions/node/v10.15.3/lib/node_modules/pnpm/lib/node_modules/signal-exit/index.js:150:5)

9:55 PM

at process.exit (internal/process/per_thread.js:157:15)

9:55 PM

at Timeout._onTimeout (/opt/nvm/versions/node/v10.15.3/lib/node_modules/pnpm/lib/err.js:10:30)

9:55 PM

at listOnTimeout (internal/timers.js:531:17)

9:55 PM

at processTimers (internal/timers.js:475:7)

9:55 PM

Error: Promise rejected with value: {

9:55 PM

errno: -28,

9:55 PM

code: 'ENOSPC',

9:55 PM

syscall: 'mkdir',

9:55 PM

path: '/rbd/pnpm-volume/store/2/registry.npmjs.org/@types/public-ip',

9:55 PM

message: 'ENOSPC: no space left on device, mkdir ' +

9:55 PM

"'/rbd/pnpm-volume/store/2/registry.npmjs.org/@types/public-ip'"

9:55 PM

}

9:55 PM

at /opt/nvm/versions/node/v10.15.3/lib/node_modules/pnpm/lib/node_modules/loud-rejection/index.js:27:12

9:55 PM

at Array.forEach (<anonymous>)

9:55 PM

at EventEmitter.<anonymous> (/opt/nvm/versions/node/v10.15.3/lib/node_modules/pnpm/lib/node_modules/loud-rejection/index.js:23:24)

9:55 PM

at EventEmitter.emit (events.js:196:13)

9:55 PM

at emit (/opt/nvm/versions/node/v10.15.3/lib/node_modules/pnpm/lib/node_modules/signal-exit/index.js:77:11)

9:55 PM

at processEmit [as emit] (/opt/nvm/versions/node/v10.15.3/lib/node_modules/pnpm/lib/node_modules/signal-exit/index.js:150:5)

9:55 PM

at process.exit (internal/process/per_thread.js:157:15)

9:55 PM

at Timeout._onTimeout (/opt/nvm/versions/node/v10.15.3/lib/node_modules/pnpm/lib/err.js:10:30)

9:55 PM

at listOnTimeout (internal/timers.js:531:17)

9:55 PM

at processTimers (internal/timers.js:475:7)

9:55 PM

Error: Promise rejected with value: {

9:55 PM

errno: -28,

9:55 PM

code: 'ENOSPC',

9:55 PM

syscall: 'mkdir',

9:55 PM

path: '/rbd/pnpm-volume/store/2/registry.npmjs.org/@types/randomstring',

9:55 PM

message: 'ENOSPC: no space left on device, mkdir ' +

9:55 PM

"'/rbd/pnpm-volume/store/2/registry.npmjs.org/@types/randomstring'"

9:55 PM

}

9:55 PM

at /opt/nvm/versions/node/v10.15.3/lib/node_modules/pnpm/lib/node_modules/loud-rejection/index.js:27:12

9:55 PM

at Array.forEach (<anonymous>)

9:55 PM

at EventEmitter.<anonymous> (/opt/nvm/versions/node/v10.15.3/lib/node_modules/pnpm/lib/node_modules/loud-rejection/index.js:23:24)

9:55 PM

at EventEmitter.emit (events.js:196:13)

9:55 PM

at emit (/opt/nvm/versions/node/v10.15.3/lib/node_modules/pnpm/lib/node_modules/signal-exit/index.js:77:11)

9:55 PM

at processEmit [as emit] (/opt/nvm/versions/node/v10.15.3/lib/node_modules/pnpm/lib/node_modules/signal-exit/index.js:150:5)

9:55 PM

at process.exit (internal/process/per_thread.js:157:15)

9:55 PM

at Timeout._onTimeout (/opt/nvm/versions/node/v10.15.3/lib/node_modules/pnpm/lib/err.js:10:30)

9:55 PM

at listOnTimeout (internal/timers.js:531:17)

9:55 PM

at processTimers (internal/timers.js:475:7)

9:55 PM

Error: Promise rejected with value: {

9:55 PM

errno: -28,

9:55 PM

code: 'ENOSPC',

9:55 PM

syscall: 'mkdir',

9:55 PM

path: '/rbd/pnpm-volume/store/2/registry.npmjs.org/@types/randomstring',

9:55 PM

message: 'ENOSPC: no space left on device, mkdir ' +

9:55 PM

"'/rbd/pnpm-volume/store/2/registry.npmjs.org/@types/randomstring'"

9:55 PM

}

9:55 PM

at /opt/nvm/versions/node/v10.15.3/lib/node_modules/pnpm/lib/node_modules/loud-rejection/index.js:27:12

9:55 PM

at Array.forEach (<anonymous>)

9:55 PM

at EventEmitter.<anonymous> (/opt/nvm/versions/node/v10.15.3/lib/node_modules/pnpm/lib/node_modules/loud-rejection/index.js:23:24)

9:55 PM

at EventEmitter.emit (events.js:196:13)

9:55 PM

at emit (/opt/nvm/versions/node/v10.15.3/lib/node_modules/pnpm/lib/node_modules/signal-exit/index.js:77:11)

9:55 PM

at processEmit [as emit] (/opt/nvm/versions/node/v10.15.3/lib/node_modules/pnpm/lib/node_modules/signal-exit/index.js:150:5)

9:55 PM

at process.exit (internal/process/per_thread.js:157:15)

9:55 PM

at Timeout._onTimeout (/opt/nvm/versions/node/v10.15.3/lib/node_modules/pnpm/lib/err.js:10:30)

9:55 PM

at listOnTimeout (internal/timers.js:531:17)

9:55 PM

at processTimers (internal/timers.js:475:7)

9:55 PM

Error: Promise rejected with value: {

9:55 PM

errno: -28,

9:55 PM

code: 'ENOSPC',

9:55 PM

syscall: 'mkdir',

9:55 PM

path: '/rbd/pnpm-volume/store/2/registry.npmjs.org/twitch-toolkit',

9:55 PM

message: 'ENOSPC: no space left on device, mkdir ' +

9:55 PM

"'/rbd/pnpm-volume/store/2/registry.npmjs.org/twitch-toolkit'"

9:55 PM

}

9:55 PM

at /opt/nvm/versions/node/v10.15.3/lib/node_modules/pnpm/lib/node_modules/loud-rejection/index.js:27:12

9:55 PM

at Array.forEach (<anonymous>)

9:55 PM

at EventEmitter.<anonymous> (/opt/nvm/versions/node/v10.15.3/lib/node_modules/pnpm/lib/node_modules/loud-rejection/index.js:23:24)

9:55 PM

at EventEmitter.emit (events.js:196:13)

9:55 PM

at emit (/opt/nvm/versions/node/v10.15.3/lib/node_modules/pnpm/lib/node_modules/signal-exit/index.js:77:11)

9:55 PM

at processEmit [as emit] (/opt/nvm/versions/node/v10.15.3/lib/node_modules/pnpm/lib/node_modules/signal-exit/index.js:150:5)

9:55 PM

at process.exit (internal/process/per_thread.js:157:15)

9:55 PM

at Timeout._onTimeout (/opt/nvm/versions/node/v10.15.3/lib/node_modules/pnpm/lib/err.js:10:30)

9:55 PM

at listOnTimeout (internal/timers.js:531:17)

9:55 PM

at processTimers (internal/timers.js:475:7)

9:55 PM

Error: Promise rejected with value: {

9:55 PM

errno: -28,

9:55 PM

code: 'ENOSPC',

9:55 PM

syscall: 'mkdir',

9:55 PM

path: '/rbd/pnpm-volume/store/2/registry.npmjs.org/twitch-toolkit',

9:55 PM

message: 'ENOSPC: no space left on device, mkdir ' +

9:55 PM

"'/rbd/pnpm-volume/store/2/registry.npmjs.org/twitch-toolkit'"

9:55 PM

}

9:55 PM

at /opt/nvm/versions/node/v10.15.3/lib/node_modules/pnpm/lib/node_modules/loud-rejection/index.js:27:12

9:55 PM

at Array.forEach (<anonymous>)

9:55 PM

at EventEmitter.<anonymous> (/opt/nvm/versions/node/v10.15.3/lib/node_modules/pnpm/lib/node_modules/loud-rejection/index.js:23:24)

9:55 PM

at EventEmitter.emit (events.js:196:13)

9:55 PM

at emit (/opt/nvm/versions/node/v10.15.3/lib/node_modules/pnpm/lib/node_modules/signal-exit/index.js:77:11)

9:55 PM

at processEmit [as emit] (/opt/nvm/versions/node/v10.15.3/lib/node_modules/pnpm/lib/node_modules/signal-exit/index.js:150:5)

9:55 PM

at process.exit (internal/process/per_thread.js:157:15)

9:55 PM

at Timeout._onTimeout (/opt/nvm/versions/node/v10.15.3/lib/node_modules/pnpm/lib/err.js:10:30)

9:55 PM

at listOnTimeout (internal/timers.js:531:17)

9:55 PM

at processTimers (internal/timers.js:475:7)

Isn’t there a command I can do to prevent this from happening ? Like some kind of no issue/error mode like on Windows OS ? That would be cool ;-;

85

Windows has a mode where it prevents all issues and errors?

but anyway: @SebPautot if the machine hosting your container is out of disk space, then I think you’d be stuck with going back to npm so that the libraries install to your app’s disk. Although we saw Daniel try that and it corrupted his project ):

1 Like
86

Is the command enable-npm ?

87

One of my projects was suspended when I tried to boost it!

I’ve created a support ticket for it but I thought it would be useful to post here as well.

the project’s name is name is ‘ricc.’

88

Sorry, there’s nothing we can do to process your issue here and the Glitch Support Team are much more responsive via email than here :slight_smile:

89

as a point of fact, yes. that’s the command I’m referring to. but I just saw this new post from the staff:

to the effect that something about NPM, something “suspension for disk space,” and something about missing code are all connected. so you gotta wonder how careful you should be about potentially corrupting your work.

90

Don’t worry I downloaded the project. Since Glitch aknowledged the issue I’m gonna wait for a fix.

91

Result of enable-pnpm after enable-npm :
node v12.0.0, with pnpm

10:24 PM

Installing...

10:24 PM

Performing headless installation

10:24 PM

Resolving: total 1, reused 0, downloaded 0

10:24 PM

Resolving: total 1, reused 0, downloaded 0

10:24 PM

Resolving: total 89, reused 60, downloaded 0

10:24 PM

Resolving: total 189, reused 60, downloaded 6

10:24 PM

Resolving: total 320, reused 60, downloaded 26

10:24 PM

Packages: +338

10:24 PM

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

10:24 PM

Resolving: total 338, reused 60, downloaded 33

10:24 PM

a minute ago

Resolving: total 338, reused 60, downloaded 33

10:24 PM

Resolving: total 338, reused 60, downloaded 33

10:24 PM

Resolving: total 338, reused 60, downloaded 33

10:24 PM

Resolving: total 338, reused 60, downloaded 33

10:24 PM

Resolving: total 338, reused 60, downloaded 33

10:24 PM

Resolving: total 338, reused 60, downloaded 33

10:24 PM

Resolving: total 338, reused 60, downloaded 33

10:24 PM

Resolving: total 338, reused 60, downloaded 33

10:24 PM

Resolving: total 338, reused 60, downloaded 35

10:24 PM

Resolving: total 338, reused 60, downloaded 35

10:24 PM

 WARN  EXDEV: cross-device link not permitted, link '/rbd/pnpm-volume/store/2/registry.npmjs.org/tweetnacl/1.0.3/node_modules/tweetnacl/LICENSE' -> '/rbd/pnpm-volume/3272d138-8172-457a-b3c7-d15a03abca8a/node_modules/.registry.npmjs.org/tweetnacl/1.0.3/node_modules/_tmp_7077_dd8cf923f497426b500b3fe8f50f0cdc/LICENSE'

10:24 PM

Falling back to copying packages from store

10:24 PM

 WARN  EXDEV: cross-device link not permitted, link '/rbd/pnpm-volume/store/2/registry.npmjs.org/accepts/1.3.7/node_modules/accepts/package.json' -> '/rbd/pnpm-volume/3272d138-8172-457a-b3c7-d15a03abca8a/node_modules/.registry.npmjs.org/accepts/1.3.7/node_modules/_tmp_7077_abcac517e79613c9fc6c16c9542d985e/package.json'

10:24 PM

Falling back to copying packages from store

10:24 PM

 WARN  EXDEV: cross-device link not permitted, link '/rbd/pnpm-volume/store/2/registry.npmjs.org/array-flatten/1.1.1/node_modules/array-flatten/package.json' -> '/rbd/pnpm-volume/3272d138-8172-457a-b3c7-d15a03abca8a/node_modules/.registry.npmjs.org/array-flatten/1.1.1/node_modules/_tmp_7077_1ff717aee0ddea98fbe531509b77fa76/package.json'

10:24 PM

Falling back to copying packages from store

10:24 PM

ERROR  ENOSPC: no space left on device, symlink '../../../long/4.0.0/node_modules/long' -> '/rbd/pnpm-volume/3272d138-8172-457a-b3c7-d15a03abca8a/node_modules/.registry.npmjs.org/discord.js/11.6.4/node_modules/long'

10:24 PM

 WARN  EXDEV: cross-device link not permitted, link '/rbd/pnpm-volume/store/2/registry.npmjs.org/content-type/1.0.4/node_modules/content-type/package.json' -> '/rbd/pnpm-volume/3272d138-8172-457a-b3c7-d15a03abca8a/node_modules/.registry.npmjs.org/content-type/1.0.4/node_modules/_tmp_7077_70bc0eecb6b8ac7b4b265670d2aebe8c/package.json'

10:24 PM

Falling back to copying packages from store

10:24 PM

 WARN  EXDEV: cross-device link not permitted, link '/rbd/pnpm-volume/store/2/registry.npmjs.org/ms/2.0.0/node_modules/ms/package.json' -> '/rbd/pnpm-volume/3272d138-8172-457a-b3c7-d15a03abca8a/node_modules/.registry.npmjs.org/ms/2.0.0/node_modules/_tmp_7077_edb63d69a37eccd2162d41eee75e510e/package.json'

10:24 PM

Falling back to copying packages from store

10:24 PM

 WARN  EXDEV: cross-device link not permitted, link '/rbd/pnpm-volume/store/2/registry.npmjs.org/async-limiter/1.0.1/node_modules/async-limiter/package.json' -> '/rbd/pnpm-volume/3272d138-8172-457a-b3c7-d15a03abca8a/node_modules/.registry.npmjs.org/async-limiter/1.0.1/node_modules/_tmp_7077_79a6cbb3241d6dfcbe5ea71879e88f7d/package.json'

10:24 PM

Falling back to copying packages from store

10:24 PM

 WARN  EXDEV: cross-device link not permitted, link '/rbd/pnpm-volume/store/2/registry.npmjs.org/bytes/3.1.0/node_modules/bytes/package.json' -> '/rbd/pnpm-volume/3272d138-8172-457a-b3c7-d15a03abca8a/node_modules/.registry.npmjs.org/bytes/3.1.0/node_modules/_tmp_7077_7ae39f7b9197e8ac52510beb8a885e7e/package.json'

10:24 PM

Falling back to copying packages from store

10:24 PM

10:24 PM

┌──────────────────────────────────────────────────┐

10:24 PM

│ pnpm update check failed │

10:24 PM

│ Try running with sudo or get access │

10:24 PM

│ to the local update config store via │

10:24 PM

│ sudo chown -R $USER:$(id -gn $USER) /app/.config │

10:24 PM

└──────────────────────────────────────────────────┘
93

Glitch is aware of this issue

1 Like
95

https://gyazo.com/082e217a587c69b71fe0efc8c8d59c9e

I Can’t Open the Terminal The Project is giving random error, when i re-open the project glitch give me always others random error.
@glitch_support

96

So, when I open up my glitch editor I get some errors in the dev console about a font.

Any ideas?

97

Can you try clicking the gear at the top right of the terminal and replacing the popup text with this:

{
  "fontSize": 16,
  "fontFamily": "monospace"
}
98

Alright, let me try!

1 Like
99

Well small problem,the console uh doesn’t load :frowning:

100

Can you send a picture of what it looks like?

101

Sure! Let me get an imgur link

102

https://salvage.is-inside.me/V4pM69g9.png there is the image

103

The other day I removed a project because I had remixed the old one and by deleting the old one it does not appear in the archive and when I try to put the old name in the new one it tells me that the name is already used by another project

104

I’m not sure you can permanently delete projects. Email [email protected]?

106

If I know that it cannot be deleted but when they are archived it allows you to reuse the name in another project but it does not appear to me in the archived projects area nor does it allow me to reuse the name and I will communicate by email to see that they respond to me

1 Like
107

Hi Glitch,
I’ve had problems lately with my discord bot hosting. Booting takes forever, and just today my bot refuses to boot with Glitch. I downloaded the code and hosted it myself and it worked. Could you please help me? I also tried emailing Glitch Support but there was no response for 3 days. Thanks.

108

I’m really sorry this has been happening!

Glitch has been having a few issues and that is probably the cause of this.

Are you getting any erorrs?

109

Easy Solution

Wait until glitch is back! https://status.glitch.com

Today i have updated a bunch of files and added like 2000 lines of code, all were removed after 1 hour, i would like to say to @Glitch_Response that, it was so poor, I had many important files

110

Hi there @Surgical_Crow!
I have had a similar issue where my project has reverted back to an older version or code has been lost. Please email [email protected] or create a ticket via Happyfox and they will restore one of their backups (they take these quite regularly - around every 30 mins to an hour I think). Please give them around 24-48 hours to respond :slight_smile:
Eddie

111

Received this after “refresh” :
node v12.0.0, with pnpm

10:50 PM

Installing...

10:50 PM

Performing headless installation

10:50 PM

 ERROR  ENOSPC: no space left on device, rename '/rbd/pnpm-volume/store/2/store.json.1210373675' -> '/rbd/pnpm-volume/store/2/store.json'

How do I access this file and change its name ?

112

My static site is failing to load.
image

113

Hi i have got problem with quick.db
SqliteError: disk I/O error
How can i fix it?

Also npm packages are auto uninstalling from my projects and sometimes i cant open Terminal (failed to start terminal).

114

Hello! I just got out of a meeting about the incident and wanted to share some additional info with you about what is going on:

  • We had launched a few changes last week that significantly altered the way projects run. This caused some unexpected side effects that we have been working on since.
  • We are making some massive changes to our infrastructure to ultimately get to a place of much higher stability for the Glitch platform. Because these changes are large and affect critical pieces of our infrastructure, we risk instability in the meantime - and this time the change we made had an unexpected impact on project start times. It was very unexpected, so it took us a while to even understand what was happening.

What we are doing about it:

  • We are undoing part of the change that we made to our infrastructure, the one that most contributed to the high project start times.
  • For our infrastructure changes going forward, we are going to try a different approach that we hope to have a more controlled effect on the system, trying as much as possible to leave existing projects alone and working on infrastructure changes in parallel.
  • All of this is working toward a faster, stabler Glitch in the coming weeks.

Until this incident is resolved, I will be posting any new information that I can share in this thread.

9 Likes
115

Did they say anything about Ping Services?

116

Wonderful! Thanks for passing the info to us! Maybe when doing infastructure changes use a different test Glitch site (a direct copy of the source code) to check if they work, like test.glitch.com, where only staff accounts can log on so it doesn’t affect other projects if something goes wrong :slight_smile:

3 Likes
117

It’s funny how changes to make Glitch more stable made Glitch less stable :joy:

2 Likes
118

Lol True It’s funny but a bit stupid

119

Thanks! I’m already seeing some improvement as the package reinstalling glitch for openradio and other projects has been fixed. Keep up the hard work, I hope to see glitch become stable in a few days!

4 Likes
120

Yeah. My index.js got replaced by dots.

121

hi @ToastyNIck the ping service is a thing that right now it’s not important for glitch, they prefear make the site always operational and fast, and that is the best choise that the team can do.

For that You should buy the Boost App on glitch.com

122

Hello,

Our team is investigating why the glitch site is not loading now. Updates will continue to be posted on status.glitch.com.

2 Likes
123

:man_facepalming: :joy:
Hope it gets fixed soon! Good luck! Now, we thought the outage was bad!

Eddie

124

Looks to be fixed now!
Edit: I got bad gateway error before and now it loads but very slow :slight_smile: