It is the same stream object being passed to multiple crawlers via userData. Wasn't sure how else to pass the stream object to the routers. I'm now trying something like this...
\nlet _router: RouterHandler<CheerioCrawlingContext<any, any>>;\n\nexport const initRouter = (writer: WritableStreamDefaultWriter<string>) => {\n if (_router== undefined) {\n _router= createCheerioRouter();\n }\n\n // Metro, Food Basics\n _router.addHandler('DETAIL', async ({ request, $, log, pushData }) => {\n ...\n\n if (writer) {\n await writer.write(JSON.stringify(product));\n }\n });\n\n\n // This is a fallback route which will handle the start URL\n // as well as the LIST labeled URLs.\n _router.addDefaultHandler(async ({ request, enqueueLinks, log }) => {\n ...\n });\n\n return _router;\n}
UPDATE: Tried it and this is working for me
","upvoteCount":0,"url":"https://github.com/apify/crawlee/discussions/2565#discussioncomment-9947894"}}}-
To give a little context I am trying to stream my scraped data back to the user via API endpoint. I'm running into an issue where I pass the WriteableStream to the userData and within the route handlers I am writing the data to the stream. The issue is the userData does not maintain the properties and functions of my object being passed in, "ERROR CheerioCrawler: Request failed and reached maximum retries. TypeError: stream.write is not a function" const { readable, writable } = new TransformStream();
const writer = writable.getWriter();
async function startCluster(writer: WritableStreamDefaultWriter<string>) {
const crawler = new CheerioCrawler(
Object.assign(
{ requestQueue: await RequestQueue.open(CRAWLERS.UNIQUE_KEY) },
crawlOptions
),
new Configuration({
defaultKeyValueStoreId: CRAWLERS.UNIQUE_KEY,
defaultDatasetId: CRAWLERS.UNIQUE_KEY,
defaultRequestQueueId: CRAWLERS.UNIQUE_KEY
}));
const crawl1 = crawler.run([
{
url: `https://www.abc.com`,
userData: {
stream: writer
}
},
]);
await crawl1;
}
export const router = createCheerioRouter();
router.addDefaultHandler(async ({ request, enqueueLinks, log }) => {
await enqueueLinks({
selector: 'details-link',
label: 'DETAIL',
userData: request.userData
});
});
router.addHandler('DETAIL', async ({ request, $, log, pushData }) => {
...
await pushData(product);
const stream = request?.userData?.stream as WritableStreamDefaultWriter<string>;
if (stream) {
await stream.write(JSON.stringify(product)); // <---- This is where I run into a problem.
//"ERROR CheerioCrawler: Request failed and reached maximum retries. TypeError: stream.write is not a function"
}
});
startCrawler(writer); |
Beta Was this translation helpful? Give feedback.
-
Maybe have a map of request.uniqueKey to your stream handles instead of adding it there? |
Beta Was this translation helpful? Give feedback.
It is the same stream object being passed to multiple crawlers via userData. Wasn't sure how else to pass the stream object to the routers. I'm now trying something like this...