Card prices report being out of date but Cardhoarder's timestamp always seems to be within an hour when I spot check it

[bakert]

MariaDB [prices]> SELECT FROM_UNIXTIME(MIN(time)) FROM cache;
+--------------------------+
| FROM_UNIXTIME(MIN(time)) |
+--------------------------+
| 2023-06-24 22:12:01      |
+--------------------------+
1 row in set (0.008 sec)

MariaDB [prices]> SELECT FROM_UNIXTIME(MAX(time)) FROM cache;
+--------------------------+
| FROM_UNIXTIME(MAX(time)) |
+--------------------------+
| 2023-08-29 04:12:01      |
+--------------------------+
1 row in set (0.007 sec)

MariaDB [prices]> SELECT NOW();
+---------------------+
| NOW()               |
+---------------------+
| 2023-08-29 10:28:27 |
+---------------------+
1 row in set (0.000 sec)

MariaDB [prices]> SELECT * FROM cache WHERE card = 'Dread Return';
ERROR 1054 (42S22): Unknown column 'card' in 'where clause'
MariaDB [prices]> SELECT * FROM cache WHERE name = 'Dread Return';
+------------+--------------+------+------+-------+----------+----------+----------+
| time       | name         | high | low  | price | week     | month    | season   |
+------------+--------------+------+------+-------+----------+----------+----------+
| 1693246321 | Dread Return |   35 |    1 |     1 | 0.401198 | 0.232911 | 0.785714 |
+------------+--------------+------+------+-------+----------+----------+----------+
1 row in set (0.010 sec)

MariaDB [prices]> SELECT FROM_UNIXTIME(time) FROM cache WHERE name = 'Dread Return';
+---------------------+
| FROM_UNIXTIME(time) |
+---------------------+
| 2023-08-29 04:12:01 |
+---------------------+
1 row in set (0.011 sec)


MariaDB [prices]> SELECT NOW();
+---------------------+
| NOW()               |
+---------------------+
| 2023-08-29 10:32:44 |
+---------------------+
1 row in set (0.000 sec)

[bakert]

[bakert]

Cardhoarder's pricefile has plenty of entries for Dread Return, and some in stock, but only at higher prices.
Cardhoarders pricefile-foil has plenty of entries for Dread Return, but NONE are in stock, so maybe it's something to do with that?

MTGOTraders has Dread Return at 1¢, but their file doesn't have a timestamp so we use the Cardhoarder timestamp as the official time of the price even though we get it from both places.

[bakert]

I ran the price grabber by hand. It appeared to work fine, only complaining about the Walking Dead cards being bogus.

It paused for a quite a long time doing the caching on the end and I wonder if that just fails some amount of the time leaving us with out of date prices until it succeeds? (It also said "Sentry is attempting to send 19 pending events" which I thought was interesting.)

It did eventually succeed on my manual run and the bot started reporting that it had up-to-date prices. We should fix whatever this is before rotation in 10ish days otherwise the integrity of rotation is a little compromised.

[bakert]

After the run by hand:

[bakert]

OK checking in a little while later the prices.log file that cron makes when it runs the prices update seems to say everything went well about 47 minutes ago.

But:

and this agrees with the db.

It isn't that the prices script isn't running because the appropriate db stuff appears in the prices.log file:

pricefiles from Cardhoarder right now say:

UPDATED 2023-08-29T00:12:01+00:00
UPDATED 2023-08-29T00:13:02+00:00

which IS a few hours ago (it's 02:51 UTC rn)

so I guess it IS the case that Cardhoarder are not keeping their file as up-to-date as they used to and that's all there is to it?

Probably nothing we want to do about that.

[bakert]

I'll keep checking over the next few days and see if anything actionable bubbles up.

[bakert]

Checking this again right now and we ran successfully 20 minutes ago but the timestamps in the Cardhoarder files are from 5 hours ago. So pretty sure this 100% a Cardhoarder thing. I have emailed Nathaniel to see if it represents any kind of issue on their side. Perhaps if this is the new normal we should switch the bot to only warning if prices >1 day old or something like that. But I'll wait and see what Nathaniel says before taking any action.

[bakert]

"We did adjust the API to produce the price file roughly every 6 hours, as unfortunately access to the price file (directly or indirectly) was being used to game the pricing system."

I guess we just change our threshold to 6 hours for being out of date and roll on

[bakert]

I changed the WARNING threshold to six hours in #11644

[bakert]

Current timestamp is UPDATED 2023-08-31T12:13:01+00:00 which is 16 hours ago. But I guess we're just at Cardhoarder's mercy here. Might up from six hours to a couple of days if it persists.